{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c014a89a-05f5-446a-bc53-dd048c6c4997",
   "metadata": {},
   "source": [
    "### 1.创新性的案例\n",
    "#### 完成课程实践的基础上，重新自己起一个Notebook，有创新点\n",
    "#### 基于LangChain+ChatGLM3的本地知识库问答\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcebc886-f10f-4eaf-b6da-123c381a4bce",
   "metadata": {},
   "source": [
    "##### 实现了基于LangChain+ChatGLM3的本地知识库问答，本地知识库支持txt、md、pdf、jpg文件。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb420814-0aaf-4244-9c4b-70e087c509d7",
   "metadata": {},
   "source": [
    "## 2. 案例体验\n",
    "🔹 本案例需使用 P100 及以上规格运行，请确保运行规格一致，可按照下图切换规格。\n",
    "\n",
    "![](https://modelarts-labs-bj4-v2.obs.cn-north-4.myhuaweicloud.com/case_zoo/chatglm3/image/1.png)\n",
    "\n",
    "🔹 点击Run in ModelArts，将会进入到ModelArts CodeLab中，这时需要你登录华为云账号，如果没有账号，则需要注册一个，且要进行实名认证，参考[《ModelArts准备工作_简易版》](https://developer.huaweicloud.com/develop/aigallery/article/detail?id=4ce709d6-eb25-4fa4-b214-e2e5d6b7919c) 即可完成账号注册和实名认证。 登录之后，等待片刻，即可进入到CodeLab的运行环境\n",
    "\n",
    "🔹 出现 Out Of Memory ，请检查是否为您的参数配置过高导致，修改参数配置，重启kernel或更换更高规格资源进行规避❗❗❗"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a579a34-88f2-4fe3-94f1-88a6597a4a13",
   "metadata": {},
   "source": [
    "### 2.1 下载模型和数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11b1bd3e-e804-49fe-b51f-2b535ddf6eb1",
   "metadata": {},
   "source": [
    "下载nltk_data数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9ac410a0-8844-47fc-b837-3a8549bc8ae8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Using MoXing-v2.1.0.5d9c87c8-5d9c87c8\n",
      "INFO:root:Using OBS-Python-SDK-3.20.9.1\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import moxing as mox\n",
    "\n",
    "work_dir = '/home/ma-user/work'\n",
    "obs_path = 'obs://dtse-models/tar-models/nltk_data.tar'\n",
    "ma_path = os.path.join(work_dir, 'nltk_data.tar')\n",
    "mox.file.copy(obs_path, ma_path)\n",
    "\n",
    "mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/case_zoo/langchain-ChatGLM/file/docs','/home/ma-user/work/docs')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf52e9db-3d99-452f-807d-38a048ac4db0",
   "metadata": {},
   "source": [
    "进入nltk_data目录，解压数据压缩包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "130fdc75-d9e6-4971-96af-74bc32f5e6f4",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ma-user/work\n",
      "nltk_data/\n",
      "nltk_data/misc/\n",
      "nltk_data/misc/mwa_ppdb.zip\n",
      "nltk_data/misc/perluniprops.xml\n",
      "nltk_data/misc/mwa_ppdb.xml\n",
      "nltk_data/misc/perluniprops.zip\n",
      "nltk_data/tokenizers/\n",
      "nltk_data/tokenizers/punkt/\n",
      "nltk_data/tokenizers/punkt/french.pickle\n",
      "nltk_data/tokenizers/punkt/polish.pickle\n",
      "nltk_data/tokenizers/punkt/.DS_Store\n",
      "nltk_data/tokenizers/punkt/portuguese.pickle\n",
      "nltk_data/tokenizers/punkt/german.pickle\n",
      "nltk_data/tokenizers/punkt/swedish.pickle\n",
      "nltk_data/tokenizers/punkt/malayalam.pickle\n",
      "nltk_data/tokenizers/punkt/estonian.pickle\n",
      "nltk_data/tokenizers/punkt/finnish.pickle\n",
      "nltk_data/tokenizers/punkt/spanish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/\n",
      "nltk_data/tokenizers/punkt/PY3/french.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/polish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/portuguese.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/german.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/swedish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/malayalam.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/estonian.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/finnish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/spanish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/danish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/greek.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/russian.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/slovene.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/norwegian.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/english.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/dutch.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/italian.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/czech.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/README\n",
      "nltk_data/tokenizers/punkt/PY3/turkish.pickle\n",
      "nltk_data/tokenizers/punkt/danish.pickle\n",
      "nltk_data/tokenizers/punkt/greek.pickle\n",
      "nltk_data/tokenizers/punkt/russian.pickle\n",
      "nltk_data/tokenizers/punkt/slovene.pickle\n",
      "nltk_data/tokenizers/punkt/norwegian.pickle\n",
      "nltk_data/tokenizers/punkt/english.pickle\n",
      "nltk_data/tokenizers/punkt/dutch.pickle\n",
      "nltk_data/tokenizers/punkt/italian.pickle\n",
      "nltk_data/tokenizers/punkt/czech.pickle\n",
      "nltk_data/tokenizers/punkt/README\n",
      "nltk_data/tokenizers/punkt/turkish.pickle\n",
      "nltk_data/tokenizers/punkt.zip\n",
      "nltk_data/tokenizers/punkt.xml\n",
      "nltk_data/sentiment/\n",
      "nltk_data/sentiment/vader_lexicon.xml\n",
      "nltk_data/sentiment/vader_lexicon.zip\n",
      "nltk_data/chunkers/\n",
      "nltk_data/chunkers/maxent_ne_chunker.xml\n",
      "nltk_data/chunkers/maxent_ne_chunker.zip\n",
      "nltk_data/help/\n",
      "nltk_data/help/tagsets.xml\n",
      "nltk_data/help/tagsets.zip\n",
      "nltk_data/taggers/\n",
      "nltk_data/taggers/averaged_perceptron_tagger.xml\n",
      "nltk_data/taggers/averaged_perceptron_tagger.zip\n",
      "nltk_data/taggers/universal_tagset.xml\n",
      "nltk_data/taggers/averaged_perceptron_tagger_ru.zip\n",
      "nltk_data/taggers/averaged_perceptron_tagger/\n",
      "nltk_data/taggers/averaged_perceptron_tagger/averaged_perceptron_tagger.pickle\n",
      "nltk_data/taggers/maxent_treebank_pos_tagger.zip\n",
      "nltk_data/taggers/universal_tagset.zip\n",
      "nltk_data/taggers/averaged_perceptron_tagger_ru.xml\n",
      "nltk_data/taggers/maxent_treebank_pos_tagger.xml\n",
      "nltk_data/corpora/\n",
      "nltk_data/corpora/cess_cat.xml\n",
      "nltk_data/corpora/twitter_samples.xml\n",
      "nltk_data/corpora/udhr.zip\n",
      "nltk_data/corpora/ieer.zip\n",
      "nltk_data/corpora/senseval.xml\n",
      "nltk_data/corpora/nonbreaking_prefixes.zip\n",
      "nltk_data/corpora/mac_morpho.xml\n",
      "nltk_data/corpora/mte_teip5.xml\n",
      "nltk_data/corpora/omw.zip\n",
      "nltk_data/corpora/reuters.xml\n",
      "nltk_data/corpora/shakespeare.zip\n",
      "nltk_data/corpora/propbank.zip\n",
      "nltk_data/corpora/rte.zip\n",
      "nltk_data/corpora/ptb3.zip\n",
      "nltk_data/corpora/kimmo.xml\n",
      "nltk_data/corpora/names.xml\n",
      "nltk_data/corpora/comparative_sentences.zip\n",
      "nltk_data/corpora/ycoe.zip\n",
      "nltk_data/corpora/nps_chat.zip\n",
      "nltk_data/corpora/sentiwordnet.zip\n",
      "nltk_data/corpora/smultron.zip\n",
      "nltk_data/corpora/lin_thesaurus.zip\n",
      "nltk_data/corpora/jeita.zip\n",
      "nltk_data/corpora/chat80.xml\n",
      "nltk_data/corpora/knbc.zip\n",
      "nltk_data/corpora/sentence_polarity.xml\n",
      "nltk_data/corpora/conll2007.zip\n",
      "nltk_data/corpora/panlex_swadesh.xml\n",
      "nltk_data/corpora/verbnet3.zip\n",
      "nltk_data/corpora/wordnet2022.zip\n",
      "nltk_data/corpora/nombank.1.0.xml\n",
      "nltk_data/corpora/comtrans.xml\n",
      "nltk_data/corpora/stopwords.xml\n",
      "nltk_data/corpora/verbnet3.xml\n",
      "nltk_data/corpora/conll2002.xml\n",
      "nltk_data/corpora/omw-1.4.zip\n",
      "nltk_data/corpora/qc.zip\n",
      "nltk_data/corpora/ppattach.xml\n",
      "nltk_data/corpora/machado.xml\n",
      "nltk_data/corpora/wordnet2021.xml\n",
      "nltk_data/corpora/stopwords.zip\n",
      "nltk_data/corpora/framenet_v15.xml\n",
      "nltk_data/corpora/ptb.zip\n",
      "nltk_data/corpora/lin_thesaurus.xml\n",
      "nltk_data/corpora/product_reviews_1.zip\n",
      "nltk_data/corpora/cess_esp.zip\n",
      "nltk_data/corpora/words.xml\n",
      "nltk_data/corpora/nps_chat.xml\n",
      "nltk_data/corpora/reuters.zip\n",
      "nltk_data/corpora/brown_tei.xml\n",
      "nltk_data/corpora/toolbox.zip\n",
      "nltk_data/corpora/swadesh.xml\n",
      "nltk_data/corpora/ppattach.zip\n",
      "nltk_data/corpora/opinion_lexicon.zip\n",
      "nltk_data/corpora/semcor.zip\n",
      "nltk_data/corpora/cmudict.xml\n",
      "nltk_data/corpora/machado.zip\n",
      "nltk_data/corpora/subjectivity.zip\n",
      "nltk_data/corpora/pl196x.zip\n",
      "nltk_data/corpora/city_database.xml\n",
      "nltk_data/corpora/treebank.zip\n",
      "nltk_data/corpora/pil.zip\n",
      "nltk_data/corpora/mac_morpho.zip\n",
      "nltk_data/corpora/masc_tagged.xml\n",
      "nltk_data/corpora/universal_treebanks_v20.zip\n",
      "nltk_data/corpora/genesis.xml\n",
      "nltk_data/corpora/indian.xml\n",
      "nltk_data/corpora/timit.xml\n",
      "nltk_data/corpora/state_union.zip\n",
      "nltk_data/corpora/framenet_v17.xml\n",
      "nltk_data/corpora/opinion_lexicon.xml\n",
      "nltk_data/corpora/problem_reports.xml\n",
      "nltk_data/corpora/wordnet.zip\n",
      "nltk_data/corpora/framenet_v17.zip\n",
      "nltk_data/corpora/subjectivity.xml\n",
      "nltk_data/corpora/sentence_polarity.zip\n",
      "nltk_data/corpora/semcor.xml\n",
      "nltk_data/corpora/city_database.zip\n",
      "nltk_data/corpora/unicode_samples.xml\n",
      "nltk_data/corpora/bcp47.xml\n",
      "nltk_data/corpora/dolch.zip\n",
      "nltk_data/corpora/ptb.xml\n",
      "nltk_data/corpora/switchboard.xml\n",
      "nltk_data/corpora/wordnet2021.zip\n",
      "nltk_data/corpora/extended_omw.zip\n",
      "nltk_data/corpora/framenet_v15.zip\n",
      "nltk_data/corpora/genesis.zip\n",
      "nltk_data/corpora/movie_reviews.zip\n",
      "nltk_data/corpora/abc.zip\n",
      "nltk_data/corpora/dependency_treebank.zip\n",
      "nltk_data/corpora/unicode_samples.zip\n",
      "nltk_data/corpora/comtrans.zip\n",
      "nltk_data/corpora/nonbreaking_prefixes.xml\n",
      "nltk_data/corpora/pl196x.xml\n",
      "nltk_data/corpora/bcp47.zip\n",
      "nltk_data/corpora/switchboard.zip\n",
      "nltk_data/corpora/pros_cons.zip\n",
      "nltk_data/corpora/wordnet_ic.zip\n",
      "nltk_data/corpora/udhr.xml\n",
      "nltk_data/corpora/dolch.xml\n",
      "nltk_data/corpora/shakespeare.xml\n",
      "nltk_data/corpora/verbnet.zip\n",
      "nltk_data/corpora/pe08.zip\n",
      "nltk_data/corpora/conll2007.xml\n",
      "nltk_data/corpora/brown.xml\n",
      "nltk_data/corpora/propbank.xml\n",
      "nltk_data/corpora/biocreative_ppi.zip\n",
      "nltk_data/corpora/product_reviews_2.xml\n",
      "nltk_data/corpora/senseval.zip\n",
      "nltk_data/corpora/unicode.notes\n",
      "nltk_data/corpora/gutenberg.zip\n",
      "nltk_data/corpora/sinica_treebank.zip\n",
      "nltk_data/corpora/state_union.xml\n",
      "nltk_data/corpora/product_reviews_2.zip\n",
      "nltk_data/corpora/ieer.xml\n",
      "nltk_data/corpora/paradigms.xml\n",
      "nltk_data/corpora/wordnet31.zip\n",
      "nltk_data/corpora/cess_esp.xml\n",
      "nltk_data/corpora/indian.zip\n",
      "nltk_data/corpora/conll2000.xml\n",
      "nltk_data/corpora/abc.xml\n",
      "nltk_data/corpora/timit.zip\n",
      "nltk_data/corpora/udhr2.xml\n",
      "nltk_data/corpora/floresta.xml\n",
      "nltk_data/corpora/inaugural.zip\n",
      "nltk_data/corpora/panlex_swadesh.zip\n",
      "nltk_data/corpora/gazetteers.xml\n",
      "nltk_data/corpora/product_reviews_1.xml\n",
      "nltk_data/corpora/movie_reviews.xml\n",
      "nltk_data/corpora/biocreative_ppi.xml\n",
      "nltk_data/corpora/webtext.xml\n",
      "nltk_data/corpora/udhr2.zip\n",
      "nltk_data/corpora/twitter_samples.zip\n",
      "nltk_data/corpora/comparative_sentences.xml\n",
      "nltk_data/corpora/knbc.xml\n",
      "nltk_data/corpora/cmudict.zip\n",
      "nltk_data/corpora/omw.xml\n",
      "nltk_data/corpora/masc_tagged.zip\n",
      "nltk_data/corpora/problem_reports.zip\n",
      "nltk_data/corpora/cess_cat.zip\n",
      "nltk_data/corpora/conll2000.zip\n",
      "nltk_data/corpora/swadesh.zip\n",
      "nltk_data/corpora/pil.xml\n",
      "nltk_data/corpora/verbnet.xml\n",
      "nltk_data/corpora/sinica_treebank.xml\n",
      "nltk_data/corpora/kimmo.zip\n",
      "nltk_data/corpora/conll2002.zip\n",
      "nltk_data/corpora/floresta.zip\n",
      "nltk_data/corpora/treebank.xml\n",
      "nltk_data/corpora/listing.csv\n",
      "nltk_data/corpora/nombank.1.0.zip\n",
      "nltk_data/corpora/listing.csv.zip\n",
      "nltk_data/corpora/words.zip\n",
      "nltk_data/corpora/webtext.zip\n",
      "nltk_data/corpora/crubadan.xml\n",
      "nltk_data/corpora/pros_cons.xml\n",
      "nltk_data/corpora/omw-1.4.xml\n",
      "nltk_data/corpora/brown_tei.zip\n",
      "nltk_data/corpora/jeita.xml\n",
      "nltk_data/corpora/ycoe.xml\n",
      "nltk_data/corpora/alpino.zip\n",
      "nltk_data/corpora/universal_treebanks_v20.xml\n",
      "nltk_data/corpora/gazetteers.zip\n",
      "nltk_data/corpora/europarl_raw.xml\n",
      "nltk_data/corpora/paradigms.zip\n",
      "nltk_data/corpora/qc.xml\n",
      "nltk_data/corpora/chat80.zip\n",
      "nltk_data/corpora/europarl_raw.zip\n",
      "nltk_data/corpora/mte_teip5.zip\n",
      "nltk_data/corpora/toolbox.xml\n",
      "nltk_data/corpora/brown.zip\n",
      "nltk_data/corpora/sentiwordnet.xml\n",
      "nltk_data/corpora/wordnet_ic.xml\n",
      "nltk_data/corpora/gutenberg.xml\n",
      "nltk_data/corpora/names.zip\n",
      "nltk_data/corpora/extended_omw.xml\n",
      "nltk_data/corpora/wordnet.xml\n",
      "nltk_data/corpora/inaugural.xml\n",
      "nltk_data/corpora/wordnet31.xml\n",
      "nltk_data/corpora/crubadan.zip\n",
      "nltk_data/corpora/wordnet2022.xml\n",
      "nltk_data/corpora/alpino.xml\n",
      "nltk_data/corpora/smultron.xml\n",
      "nltk_data/corpora/pe08.xml\n",
      "nltk_data/corpora/rte.xml\n",
      "nltk_data/corpora/dependency_treebank.xml\n",
      "nltk_data/stemmers/\n",
      "nltk_data/stemmers/snowball_data.xml\n",
      "nltk_data/stemmers/rslp.xml\n",
      "nltk_data/stemmers/snowball_data.zip\n",
      "nltk_data/stemmers/porter_test.xml\n",
      "nltk_data/stemmers/porter_test.zip\n",
      "nltk_data/stemmers/rslp.zip\n",
      "nltk_data/grammars/\n",
      "nltk_data/grammars/book_grammars.zip\n",
      "nltk_data/grammars/book_grammars.xml\n",
      "nltk_data/grammars/spanish_grammars.xml\n",
      "nltk_data/grammars/sample_grammars.xml\n",
      "nltk_data/grammars/large_grammars.xml\n",
      "nltk_data/grammars/large_grammars.zip\n",
      "nltk_data/grammars/basque_grammars.xml\n",
      "nltk_data/grammars/spanish_grammars.zip\n",
      "nltk_data/grammars/sample_grammars.zip\n",
      "nltk_data/grammars/basque_grammars.zip\n",
      "nltk_data/models/\n",
      "nltk_data/models/word2vec_sample.xml\n",
      "nltk_data/models/moses_sample.zip\n",
      "nltk_data/models/moses_sample.xml\n",
      "nltk_data/models/bllip_wsj_no_aux.xml\n",
      "nltk_data/models/word2vec_sample.zip\n",
      "nltk_data/models/bllip_wsj_no_aux.zip\n",
      "nltk_data/models/wmt15_eval.zip\n",
      "nltk_data/models/wmt15_eval.xml\n"
     ]
    }
   ],
   "source": [
    "os.chdir(work_dir)\n",
    "!pwd\n",
    "!tar -xvf nltk_data.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50ea0726-14f2-440f-a08f-e59451ed373d",
   "metadata": {},
   "source": [
    "下载text2vec-large-chinese模型，用于中文通用语义匹配"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "545bced4-7f04-4a86-aaaa-074e4c84b1e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import moxing as mox\n",
    "\n",
    "obs_path = 'obs://dtse-models/tar-models/text2vec-large-chinese.tar'\n",
    "ma_path = os.path.join(work_dir, 'text2vec-large-chinese.tar')\n",
    "mox.file.copy(obs_path, ma_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e7dcb42-884c-44ca-8ece-0d6f3935133e",
   "metadata": {},
   "source": [
    "进入text2vec-large-chinese目录，解压模型压缩包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "92f659f9-dadc-4bf6-9800-338f4a8ec578",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ma-user/work\n",
      "text2vec-large-chinese/\n",
      "text2vec-large-chinese/.gitattributes\n",
      "text2vec-large-chinese/README.md\n",
      "text2vec-large-chinese/config.json\n",
      "text2vec-large-chinese/eval_results.txt\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/blobs/\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/blobs/eaf5cb71c0eeab7db3c5171da504e5867b3f67a78e07bdba9b52d334ae35adb3.lock\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/refs/\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/refs/main\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/snapshots/\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/snapshots/064717f2acfd7253bea91079d59b82e50b58c886/\n",
      "text2vec-large-chinese/pytorch_model.bin\n",
      "text2vec-large-chinese/special_tokens_map.json\n",
      "text2vec-large-chinese/tmpqlu9nxcm\n",
      "text2vec-large-chinese/tokenizer.json\n",
      "text2vec-large-chinese/tokenizer_config.json\n",
      "text2vec-large-chinese/vocab.txt\n"
     ]
    }
   ],
   "source": [
    "os.chdir(work_dir)\n",
    "!pwd\n",
    "!tar -xvf text2vec-large-chinese.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a5bc17f-c539-45d9-b80c-d48ebcecea0b",
   "metadata": {},
   "source": [
    "### 2.2 配置环境\n",
    "\n",
    "本案例依赖Python3.10.10及以上环境，因此我们首先创建虚拟环境："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "86ab2994-2d9c-4404-a017-c5a3b8a8d671",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!\n",
      "  RequestsDependencyWarning)\n",
      "Collecting package metadata (current_repodata.json): done\n",
      "Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.\n",
      "Collecting package metadata (repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/ma-user/anaconda3/envs/python-3.10.10\n",
      "\n",
      "  added / updated specs:\n",
      "    - python=3.10.10\n",
      "\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    _libgcc_mutex-0.1          |             main           3 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    _openmp_mutex-5.1          |            1_gnu          21 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    bzip2-1.0.8                |       h7b6447c_0          78 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    ca-certificates-2023.12.12 |       h06a4308_0         126 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    ld_impl_linux-64-2.38      |       h1181459_1         654 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    libffi-3.4.4               |       h6a678d5_0         142 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    libgcc-ng-11.2.0           |       h1234567_1         5.3 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    libgomp-11.2.0             |       h1234567_1         474 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    libuuid-1.41.5             |       h5eee18b_0          27 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    ncurses-6.4                |       h6a678d5_0         914 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    openssl-1.1.1w             |       h7f8727e_0         3.7 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    pip-23.3.1                 |  py310h06a4308_0         2.7 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    python-3.10.10             |       h7a1cb2a_2        26.9 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    readline-8.2               |       h5eee18b_0         357 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    setuptools-68.2.2          |  py310h06a4308_0         957 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    sqlite-3.41.2              |       h5eee18b_0         1.2 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    tk-8.6.12                  |       h1ccaba5_0         3.0 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    tzdata-2023c               |       h04d1e81_0         116 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    wheel-0.41.2               |  py310h06a4308_0         109 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    xz-5.4.5                   |       h5eee18b_0         646 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    zlib-1.2.13                |       h5eee18b_0         103 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:        52.2 MB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  _libgcc_mutex      anaconda/pkgs/main/linux-64::_libgcc_mutex-0.1-main\n",
      "  _openmp_mutex      anaconda/pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu\n",
      "  bzip2              anaconda/pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0\n",
      "  ca-certificates    anaconda/pkgs/main/linux-64::ca-certificates-2023.12.12-h06a4308_0\n",
      "  ld_impl_linux-64   anaconda/pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1\n",
      "  libffi             anaconda/pkgs/main/linux-64::libffi-3.4.4-h6a678d5_0\n",
      "  libgcc-ng          anaconda/pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1\n",
      "  libgomp            anaconda/pkgs/main/linux-64::libgomp-11.2.0-h1234567_1\n",
      "  libstdcxx-ng       anaconda/pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1\n",
      "  libuuid            anaconda/pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0\n",
      "  ncurses            anaconda/pkgs/main/linux-64::ncurses-6.4-h6a678d5_0\n",
      "  openssl            anaconda/pkgs/main/linux-64::openssl-1.1.1w-h7f8727e_0\n",
      "  pip                anaconda/pkgs/main/linux-64::pip-23.3.1-py310h06a4308_0\n",
      "  python             anaconda/pkgs/main/linux-64::python-3.10.10-h7a1cb2a_2\n",
      "  readline           anaconda/pkgs/main/linux-64::readline-8.2-h5eee18b_0\n",
      "  setuptools         anaconda/pkgs/main/linux-64::setuptools-68.2.2-py310h06a4308_0\n",
      "  sqlite             anaconda/pkgs/main/linux-64::sqlite-3.41.2-h5eee18b_0\n",
      "  tk                 anaconda/pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0\n",
      "  tzdata             anaconda/pkgs/main/noarch::tzdata-2023c-h04d1e81_0\n",
      "  wheel              anaconda/pkgs/main/linux-64::wheel-0.41.2-py310h06a4308_0\n",
      "  xz                 anaconda/pkgs/main/linux-64::xz-5.4.5-h5eee18b_0\n",
      "  zlib               anaconda/pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0\n",
      "\n",
      "\n",
      "\n",
      "Downloading and Extracting Packages\n",
      "libgomp-11.2.0       | 474 KB    | ##################################### | 100% \n",
      "libgcc-ng-11.2.0     | 5.3 MB    | ##################################### | 100% \n",
      "wheel-0.41.2         | 109 KB    | ##################################### | 100% \n",
      "libuuid-1.41.5       | 27 KB     | ##################################### | 100% \n",
      "bzip2-1.0.8          | 78 KB     | ##################################### | 100% \n",
      "libffi-3.4.4         | 142 KB    | ##################################### | 100% \n",
      "pip-23.3.1           | 2.7 MB    | ##################################### | 100% \n",
      "_openmp_mutex-5.1    | 21 KB     | ##################################### | 100% \n",
      "tzdata-2023c         | 116 KB    | ##################################### | 100% \n",
      "ncurses-6.4          | 914 KB    | ##################################### | 100% \n",
      "tk-8.6.12            | 3.0 MB    | ##################################### | 100% \n",
      "libstdcxx-ng-11.2.0  | 4.7 MB    | ##################################### | 100% \n",
      "sqlite-3.41.2        | 1.2 MB    | ##################################### | 100% \n",
      "setuptools-68.2.2    | 957 KB    | ##################################### | 100% \n",
      "ld_impl_linux-64-2.3 | 654 KB    | ##################################### | 100% \n",
      "_libgcc_mutex-0.1    | 3 KB      | ##################################### | 100% \n",
      "ca-certificates-2023 | 126 KB    | ##################################### | 100% \n",
      "xz-5.4.5             | 646 KB    | ##################################### | 100% \n",
      "python-3.10.10       | 26.9 MB   | ##################################### | 100% \n",
      "zlib-1.2.13          | 103 KB    | ##################################### | 100% \n",
      "readline-8.2         | 357 KB    | ##################################### | 100% \n",
      "openssl-1.1.1w       | 3.7 MB    | ##################################### | 100% \n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "#\n",
      "# To activate this environment, use\n",
      "#\n",
      "#     $ conda activate python-3.10.10\n",
      "#\n",
      "# To deactivate an active environment, use\n",
      "#\n",
      "#     $ conda deactivate\n",
      "\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting ipykernel\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/5d/4b/ffb537e392e730c9a5b02758f9c87077d9087bcb0d957853e13f121e5ea7/ipykernel-6.23.1-py3-none-any.whl (152 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.2/152.2 kB\u001b[0m \u001b[31m13.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting comm>=0.1.1 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/74/f3/b88d7e1dadf741550c56b70d7ce62673354fddb68e143d193ceb80224208/comm-0.1.3-py3-none-any.whl (6.6 kB)\n",
      "Collecting debugpy>=1.6.5 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/1f/19/345c21f6b62acf556c39e4358a22b0ad868fecb462c1041c13513d229b33/debugpy-1.6.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m29.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting ipython>=7.23.1 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/09/db/d641ee07f319002393524b6c5a8b47370520dcb2b6166a0972cfe9398c60/ipython-8.10.0-py3-none-any.whl (784 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m784.3/784.3 kB\u001b[0m \u001b[31m18.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting jupyter-client>=6.1.12 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/07/37/4019d2c41ca333c08dfdfeb84c0fc0368c8defbbd3c8f0c9a530851e5813/jupyter_client-8.2.0-py3-none-any.whl (103 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m103.2/103.2 kB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting jupyter-core!=5.0.*,>=4.12 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/41/1e/92a67f333b9335f04ce409799c030dcfb291712658b9d9d13997f7c91e5a/jupyter_core-5.3.0-py3-none-any.whl (93 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.2/93.2 kB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting matplotlib-inline>=0.1 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/a6/2d/2230afd570c70074e80fd06857ba2bdc5f10c055bd9125665fe276fadb67/matplotlib_inline-0.1.3-py3-none-any.whl (8.2 kB)\n",
      "Collecting nest-asyncio (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/06/e0/93453ebab12f5ce9a9ceda2ff71648b30e5f2ce5bba19ee3c95cbd0aaa67/nest_asyncio-1.5.4-py3-none-any.whl (5.1 kB)\n",
      "Collecting packaging (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ec/1a/610693ac4ee14fcdf2d9bf3c493370e4f2ef7ae2e19217d7a237ff42367d/packaging-23.2-py3-none-any.whl (53 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.0/53.0 kB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting psutil (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/6e/c8/784968329c1c67c28cce91991ef9af8a8913aa5a3399a6a8954b1380572f/psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m280.2/280.2 kB\u001b[0m \u001b[31m29.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pyzmq>=20 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/79/f4/c33ff6e3d7bfbceecbb2176f75328c897365a519f507d226e44eea74d6d2/pyzmq-25.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m21.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting tornado>=6.1 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/66/a5/e6da56c03ff61200d5a43cfb75ab09316fc0836aa7ee26b4e9dcbfc3ae85/tornado-6.3.3-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m427.7/427.7 kB\u001b[0m \u001b[31m20.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting traitlets>=5.4.0 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/77/75/c28e9ef7abec2b7e9ff35aea3e0be6c1aceaf7873c26c95ae1f0d594de71/traitlets-5.9.0-py3-none-any.whl (117 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m117.4/117.4 kB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting backcall (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/4c/1c/ff6546b6c12603d8dd1070aa3c3d273ad4c07f5771689a7b69a550e8c951/backcall-0.2.0-py2.py3-none-any.whl (11 kB)\n",
      "Collecting decorator (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/d5/50/83c593b07763e1161326b3b8c6686f0f4b0f24d5526546bee538c89837d6/decorator-5.1.1-py3-none-any.whl (9.1 kB)\n",
      "Collecting jedi>=0.16 (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/6d/60/4acda63286ef6023515eb914543ba36496b8929cb7af49ecce63afde09c6/jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m26.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting pickleshare (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/9a/41/220f49aaea88bc6fa6cba8d05ecf24676326156c23b991e80b3f2fc24c77/pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)\n",
      "Collecting prompt-toolkit<3.1.0,>=3.0.30 (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/87/3f/1f5a0ff475ae6481f4b0d45d4d911824d3218b94ee2a97a8cb84e5569836/prompt_toolkit-3.0.38-py3-none-any.whl (385 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m385.8/385.8 kB\u001b[0m \u001b[31m27.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pygments>=2.4.0 (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/0b/42/d9d95cc461f098f204cd20c85642ae40fbff81f74c300341b8d0e0df14e0/Pygments-2.14.0-py3-none-any.whl (1.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m55.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting stack-data (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/6a/81/aa96c25c27f78cdc444fec27d80f4c05194c591465e491a1358d8a035bc1/stack_data-0.6.2-py3-none-any.whl (24 kB)\n",
      "Collecting pexpect>4.3 (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/39/7b/88dbb785881c28a102619d46423cb853b46dbccc70d3ac362d99773a78ce/pexpect-4.8.0-py2.py3-none-any.whl (59 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.0/59.0 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting python-dateutil>=2.8.2 (from jupyter-client>=6.1.12->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m247.7/247.7 kB\u001b[0m \u001b[31m15.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting platformdirs>=2.5 (from jupyter-core!=5.0.*,>=4.12->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ba/24/a83a900a90105f8ad3f20df5bb5a2cde886df7125c7827e196e4ed4fa8a7/platformdirs-3.0.0-py3-none-any.whl (14 kB)\n",
      "Collecting parso<0.9.0,>=0.8.0 (from jedi>=0.16->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/05/63/8011bd08a4111858f79d2b09aad86638490d62fbf881c44e434a6dfca87b/parso-0.8.3-py2.py3-none-any.whl (100 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100.8/100.8 kB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting ptyprocess>=0.5 (from pexpect>4.3->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)\n",
      "Collecting wcwidth (from prompt-toolkit<3.1.0,>=3.0.30->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/59/7c/e39aca596badaf1b78e8f547c807b04dae603a433d3e7a7e04d67f2ef3e5/wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)\n",
      "Collecting six>=1.5 (from python-dateutil>=2.8.2->jupyter-client>=6.1.12->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (11 kB)\n",
      "Collecting executing>=1.2.0 (from stack-data->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/28/3c/bc3819dd8b1a1588c9215a87271b6178cc5498acaa83885211f5d4d9e693/executing-1.2.0-py2.py3-none-any.whl (24 kB)\n",
      "Collecting asttokens>=2.1.0 (from stack-data->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/f3/e1/64679d9d0759db5b182222c81ff322c2fe2c31e156a59afd6e9208c960e5/asttokens-2.2.1-py2.py3-none-any.whl (26 kB)\n",
      "Collecting pure-eval (from stack-data->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/2b/27/77f9d5684e6bce929f5cfe18d6cfbe5133013c06cb2fbf5933670e60761d/pure_eval-0.2.2-py3-none-any.whl (11 kB)\n",
      "Installing collected packages: wcwidth, pure-eval, ptyprocess, pickleshare, executing, backcall, traitlets, tornado, six, pyzmq, pygments, psutil, prompt-toolkit, platformdirs, pexpect, parso, packaging, nest-asyncio, decorator, debugpy, python-dateutil, matplotlib-inline, jupyter-core, jedi, comm, asttokens, stack-data, jupyter-client, ipython, ipykernel\n",
      "Successfully installed asttokens-2.2.1 backcall-0.2.0 comm-0.1.3 debugpy-1.6.6 decorator-5.1.1 executing-1.2.0 ipykernel-6.23.1 ipython-8.10.0 jedi-0.18.2 jupyter-client-8.2.0 jupyter-core-5.3.0 matplotlib-inline-0.1.3 nest-asyncio-1.5.4 packaging-23.2 parso-0.8.3 pexpect-4.8.0 pickleshare-0.7.5 platformdirs-3.0.0 prompt-toolkit-3.0.38 psutil-5.9.4 ptyprocess-0.7.0 pure-eval-0.2.2 pygments-2.14.0 python-dateutil-2.8.2 pyzmq-25.1.1 six-1.16.0 stack-data-0.6.2 tornado-6.3.3 traitlets-5.9.0 wcwidth-0.2.5\n"
     ]
    }
   ],
   "source": [
    "!/home/ma-user/anaconda3/bin/conda create -n python-3.10.10 python=3.10.10 -y --override-channels --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
    "!/home/ma-user/anaconda3/envs/python-3.10.10/bin/pip install ipykernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "738f430c-401d-48d2-b330-2eca3a2a629c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "\n",
    "data = {\n",
    "   \"display_name\": \"python-3.10.10\",\n",
    "   \"env\": {\n",
    "      \"PATH\": \"/home/ma-user/anaconda3/envs/python-3.10.10/bin:/home/ma-user/anaconda3/envs/python-3.7.10/bin:/modelarts/authoring/notebook-conda/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/anaconda3/envs/PyTorch-1.8/bin\"\n",
    "   },\n",
    "   \"language\": \"python\",\n",
    "   \"argv\": [\n",
    "      \"/home/ma-user/anaconda3/envs/python-3.10.10/bin/python\",\n",
    "      \"-m\",\n",
    "      \"ipykernel\",\n",
    "      \"-f\",\n",
    "      \"{connection_file}\"\n",
    "   ]\n",
    "}\n",
    "\n",
    "if not os.path.exists(\"/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/\"):\n",
    "    os.mkdir(\"/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/\")\n",
    "\n",
    "with open('/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/kernel.json', 'w') as f:\n",
    "    json.dump(data, f, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39d8b96b-98bc-489f-becc-998c27b994dc",
   "metadata": {},
   "source": [
    "创建完成后，稍等片刻，或刷新页面，点击右上角kernel选择python-3.10.10\n",
    "\n",
    "![](https://modelarts-labs-bj4-v2.obs.cn-north-4.myhuaweicloud.com/case_zoo/chatglm3/image/2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "283d52cf-12c6-4330-ab15-40f3c0f2493e",
   "metadata": {},
   "source": [
    "### 2.3 安装依赖库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6840d348-a071-42d0-b96f-4e6ecff9547d",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting transformers==4.30.2\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/5b/0b/e45d26ccd28568013523e04f325432ea88a442b4e3020b757cf4361f0120/transformers-4.30.2-py3-none-any.whl (7.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.2/7.2 MB\u001b[0m \u001b[31m94.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting filelock (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/00/45/ec3407adf6f6b5bf867a4462b2b0af27597a26bd3cd6e2534cb6ab029938/filelock-3.12.2-py3-none-any.whl (10 kB)\n",
      "Collecting huggingface-hub<1.0,>=0.14.1 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/7f/c4/adcbe9a696c135578cabcbdd7331332daad4d49b7c43688bc2d36b3a47d2/huggingface_hub-0.16.4-py3-none-any.whl (268 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m27.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting numpy>=1.17 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/71/3c/3b1981c6a1986adc9ee7db760c0c34ea5b14ac3da9ecfcf1ea2a4ec6c398/numpy-1.25.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m18.2/18.2 MB\u001b[0m \u001b[31m84.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (23.2)\n",
      "Collecting pyyaml>=5.1 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/29/61/bf33c6c85c55bc45a29eee3195848ff2d518d84735eb0e2d8cb42e0d285e/PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m705.5/705.5 kB\u001b[0m \u001b[31m19.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting regex!=2019.12.17 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/7c/81/b064cc2c67ca2182137641f9d3fd47fe470f1a84674d9b9f91fd39bf0e6f/regex-2023.5.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (769 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m769.7/769.7 kB\u001b[0m \u001b[31m18.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting requests (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl (62 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.6/62.6 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/94/60/ff26cce378023624ffcad91edaa4871f561d6ba7295185c45037ddba80e2/tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m37.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting safetensors>=0.3.1 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/6c/f0/c17bbdb1e5f9dab29d44cade445135789f75f8f08ea2728d04493ea8412b/safetensors-0.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m48.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hCollecting tqdm>=4.27 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/00/e5/f12a80907d0884e6dff9c16d0c0114d81b8cd07dc3ae54c5e962cc83037e/tqdm-4.66.1-py3-none-any.whl (78 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.3/78.3 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting fsspec (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/37/57/eb7c3c10b187d3b8565946772ce0229c79e3c623010eda0aeb5032ff56f4/fsspec-2022.11.0-py3-none-any.whl (139 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m139.5/139.5 kB\u001b[0m \u001b[31m18.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting typing-extensions>=3.7.4.3 (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/24/21/7d397a4b7934ff4028987914ac1044d3b7d52712f30e2ac7a2ae5bc86dd0/typing_extensions-4.8.0-py3-none-any.whl (31 kB)\n",
      "Collecting charset-normalizer<4,>=2 (from requests->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/da/f1/3702ba2a7470666a62fd81c58a4c40be00670e5006a67f4d626e57f013ae/charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m142.1/142.1 kB\u001b[0m \u001b[31m10.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting idna<4,>=2.5 (from requests->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl (61 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.5/61.5 kB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting urllib3<3,>=1.21.1 (from requests->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/96/94/c31f58c7a7f470d5665935262ebd7455c7e4c7782eb525658d3dbf4b9403/urllib3-2.1.0-py3-none-any.whl (104 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m104.6/104.6 kB\u001b[0m \u001b[31m11.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting certifi>=2017.4.17 (from requests->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/64/62/428ef076be88fa93716b576e4a01f919d25968913e817077a386fcbe4f42/certifi-2023.11.17-py3-none-any.whl (162 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m162.5/162.5 kB\u001b[0m \u001b[31m13.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: tokenizers, safetensors, urllib3, typing-extensions, tqdm, regex, pyyaml, numpy, idna, fsspec, filelock, charset-normalizer, certifi, requests, huggingface-hub, transformers\n",
      "Successfully installed certifi-2023.11.17 charset-normalizer-3.3.2 filelock-3.12.2 fsspec-2022.11.0 huggingface-hub-0.16.4 idna-3.4 numpy-1.25.2 pyyaml-6.0.1 regex-2023.5.5 requests-2.31.0 safetensors-0.3.3 tokenizers-0.13.3 tqdm-4.66.1 transformers-4.30.2 typing-extensions-4.8.0 urllib3-2.1.0\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting sentencepiece==0.1.99\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/7f/e5/323dc813b3e1339305f888d035e2f3725084fc4dcf051995b366dd26cc90/sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m21.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: sentencepiece\n",
      "Successfully installed sentencepiece-0.1.99\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting torch==2.0.1\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/8c/4d/17e07377c9c3d1a0c4eb3fde1c7c16b5a0ce6133ddbabc08ceef6b7f2645/torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m619.9/619.9 MB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: filelock in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (3.12.2)\n",
      "Requirement already satisfied: typing-extensions in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (4.8.0)\n",
      "Collecting sympy (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/d2/05/e6600db80270777c4a64238a98d442f0fd07cc8915be2a1c16da7f2b9e74/sympy-1.12-py3-none-any.whl (5.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.7/5.7 MB\u001b[0m \u001b[31m99.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting networkx (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/42/31/d2f89f1ae42718f8c8a9e440ebe38d7d5fe1e0d9eb9178ce779e365b3ab0/networkx-2.8.8-py3-none-any.whl (2.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m77.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting jinja2 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/bc/c3/f068337a370801f372f2f8f6bad74a5c140f6fda3d9de154052708dd3c65/Jinja2-3.1.2-py3-none-any.whl (133 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m133.1/133.1 kB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cuda-nvrtc-cu11==11.7.99 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ea/8d/0709ba16c2831c17ec1c2ea1eeb89ada11ffa8d966d773cce0a7463b22bb/nvidia_cuda_nvrtc_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (21.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.0/21.0 MB\u001b[0m \u001b[31m85.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cuda-runtime-cu11==11.7.99 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/36/92/89cf558b514125d2ebd8344dd2f0533404b416486ff681d5434a5832a019/nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m849.3/849.3 kB\u001b[0m \u001b[31m20.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cuda-cupti-cu11==11.7.101 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/e6/9d/dd0cdcd800e642e3c82ee3b5987c751afd4f3fb9cc2752517f42c3bc6e49/nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.8/11.8 MB\u001b[0m \u001b[31m101.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cudnn-cu11==8.5.0.96 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/dc/30/66d4347d6e864334da5bb1c7571305e501dcb11b9155971421bb7bb5315f/nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m557.1/557.1 MB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cublas-cu11==11.10.3.66 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ce/41/fdeb62b5437996e841d83d7d2714ca75b886547ee8017ee2fe6ea409d983/nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m317.1/317.1 MB\u001b[0m \u001b[31m15.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cufft-cu11==10.9.0.58 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/74/79/b912a77e38e41f15a0581a59f5c3548d1ddfdda3225936fb67c342719e7a/nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m168.4/168.4 MB\u001b[0m \u001b[31m26.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-curand-cu11==10.2.10.91 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/8f/11/af78d54b2420e64a4dd19e704f5bb69dcb5a6a3138b4465d6a48cdf59a21/nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.6/54.6 MB\u001b[0m \u001b[31m60.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cusolver-cu11==11.4.0.1 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/25/4b/272f9aa7838e545b47878e4aec4f09b0fecf17dbd312cf5c5dc398b0637f/nvidia_cusolver_cu11-11.4.0.1-py3-none-manylinux1_x86_64.whl (102.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m102.6/102.6 MB\u001b[0m \u001b[31m38.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cusparse-cu11==11.7.4.91 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ea/6f/6d032cc1bb7db88a989ddce3f4968419a7edeafda362847f42f614b1f845/nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m173.2/173.2 MB\u001b[0m \u001b[31m26.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-nccl-cu11==2.14.3 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/55/92/914cdb650b6a5d1478f83148597a25e90ea37d739bd563c5096b0e8a5f43/nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m177.1/177.1 MB\u001b[0m \u001b[31m25.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-nvtx-cu11==11.7.91 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/23/d5/09493ff0e64fd77523afbbb075108f27a13790479efe86b9ffb4587671b5/nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.6/98.6 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting triton==2.0.0 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ca/31/ff6be541195daf77aa5c72303b2354661a69e717967d44d91eb4f3fdce32/triton-2.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m63.3/63.3 MB\u001b[0m \u001b[31m54.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: setuptools in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (68.2.2)\n",
      "Requirement already satisfied: wheel in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (0.41.2)\n",
      "Collecting cmake (from triton==2.0.0->torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/2e/51/3a4672a819b4532a378bfefad8f886cfe71057556e0d4eefb64523fd370a/cmake-3.27.2-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (26.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m26.1/26.1 MB\u001b[0m \u001b[31m74.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting lit (from triton==2.0.0->torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/63/ab/e4c5f26400970a3ca356d19e592af3a5f05a73d5f35470accb6600f84045/lit-16.0.5.tar.gz (138 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.0/138.0 kB\u001b[0m \u001b[31m17.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting MarkupSafe>=2.0 (from jinja2->torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/12/b3/d9ed2c0971e1435b8a62354b18d3060b66c8cb1d368399ec0b9baa7c0ee5/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)\n",
      "Collecting mpmath>=0.19 (from sympy->torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl (536 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.2/536.2 kB\u001b[0m \u001b[31m19.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hBuilding wheels for collected packages: lit\n",
      "  Building wheel for lit (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for lit: filename=lit-16.0.5-py3-none-any.whl size=88176 sha256=310bd7534f13fbe5ff5c6a21ce1ac031286252eba718b96f1ca5307c5a1e07a7\n",
      "  Stored in directory: /home/ma-user/.cache/pip/wheels/e1/dc/f7/0eb9b51ba2fcb3a439d56ce1259dc815e5bcdd329c2b874f34\n",
      "Successfully built lit\n",
      "Installing collected packages: mpmath, lit, cmake, sympy, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-cusolver-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, networkx, MarkupSafe, nvidia-cudnn-cu11, jinja2, triton, torch\n",
      "Successfully installed MarkupSafe-2.1.3 cmake-3.27.2 jinja2-3.1.2 lit-16.0.5 mpmath-1.3.0 networkx-2.8.8 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 sympy-1.12 torch-2.0.1 triton-2.0.0\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting markdown==3.4.3\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/9a/a1/1352b0e5a3c71a79fa9265726e2217f69df9fd4de0bcb5725cc61f62a5df/Markdown-3.4.3-py3-none-any.whl (93 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.9/93.9 kB\u001b[0m \u001b[31m11.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: markdown\n",
      "Successfully installed markdown-3.4.3\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting faiss-gpu==1.7.2\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/a8/71/623896382d90a9a99adf3438aa2c575535ba37804be9701d66f3337afd83/faiss_gpu-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (85.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m85.5/85.5 MB\u001b[0m \u001b[31m43.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: faiss-gpu\n",
      "Successfully installed faiss-gpu-1.7.2\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting langchain==0.0.329\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/42/4e/86204994aeb2e4ac367a7fade896b13532eae2430299052eb2c80ca35d2c/langchain-0.0.329-py3-none-any.whl (2.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m4.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: PyYAML>=5.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langchain==0.0.329) (6.0.1)\n",
      "Collecting SQLAlchemy<3,>=1.4 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/aa/1c/0b66318368b1c9ef51c5c8560530b8ef842164e10eea08cacb06739388e0/SQLAlchemy-2.0.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m58.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting aiohttp<4.0.0,>=3.8.3 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2f/16/50441c4baa39e5426181c6f630203ab65029f9a9c55d0a1019a31c26d702/aiohttp-3.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m60.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting anyio<4.0 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl (80 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m80.9/80.9 kB\u001b[0m \u001b[31m10.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting async-timeout<5.0.0,>=4.0.0 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a7/fa/e01228c2938de91d47b307831c62ab9e4001e747789d0b05baf779a6488c/async_timeout-4.0.3-py3-none-any.whl (5.7 kB)\n",
      "Collecting dataclasses-json<0.7,>=0.5.7 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/53/8c006de775834cd4ea64a445402dc195caeebb77dc76b7defb9b3887cb0d/dataclasses_json-0.6.3-py3-none-any.whl (28 kB)\n",
      "Collecting jsonpatch<2.0,>=1.33 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/73/07/02e16ed01e04a374e644b575638ec7987ae846d25ad97bcc9945a3ee4b0e/jsonpatch-1.33-py2.py3-none-any.whl (12 kB)\n",
      "Collecting langsmith<0.1.0,>=0.0.52 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c0/a2/7814b2341d2919f8305cdaff2e37f76b04c45839f402a38cf13ef7153bea/langsmith-0.0.75-py3-none-any.whl (46 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.7/46.7 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: numpy<2,>=1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langchain==0.0.329) (1.25.2)\n",
      "Collecting pydantic<3,>=1 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/dd/b7/9aea7ee6c01fe3f3c03b8ca3c7797c866df5fecece9d6cb27caa138db2e2/pydantic-2.5.3-py3-none-any.whl (381 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m381.9/381.9 kB\u001b[0m \u001b[31m34.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: requests<3,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langchain==0.0.329) (2.31.0)\n",
      "Collecting tenacity<9.0.0,>=8.1.0 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f4/f1/990741d5bb2487d529d20a433210ffa136a367751e454214013b441c4575/tenacity-8.2.3-py3-none-any.whl (24 kB)\n",
      "Collecting attrs>=17.3.0 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f0/eb/fcb708c7bf5056045e9e98f62b93bd7467eb718b0202e7698eb11d66416c/attrs-23.1.0-py3-none-any.whl (61 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.2/61.2 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting multidict<7.0,>=4.5 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/56/b5/ac112889bfc68e6cf4eda1e4325789b166c51c6cd29d5633e28fb2c2f966/multidict-6.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.5/114.5 kB\u001b[0m \u001b[31m13.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting yarl<2.0,>=1.0 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c3/a0/0ade1409d184cbc9e85acd403a386a7c0563b92ff0f26d138ff9e86e48b4/yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m301.6/301.6 kB\u001b[0m \u001b[31m26.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting frozenlist>=1.1.1 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ec/25/0c87df2e53c0c5d90f7517ca0ff7aca78d050a8ec4d32c4278e8c0e52e51/frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m239.5/239.5 kB\u001b[0m \u001b[31m19.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting aiosignal>=1.1.2 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/76/ac/a7305707cb852b7e16ff80eaf5692309bde30e2b1100a1fcacdc8f731d97/aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
      "Requirement already satisfied: idna>=2.8 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from anyio<4.0->langchain==0.0.329) (3.4)\n",
      "Collecting sniffio>=1.1 (from anyio<4.0->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c3/a0/5dba8ed157b0136607c7f2151db695885606968d1fae123dc3391e0cfdbf/sniffio-1.3.0-py3-none-any.whl (10 kB)\n",
      "Collecting exceptiongroup (from anyio<4.0->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b8/9a/5028fd52db10e600f1c4674441b968cf2ea4959085bfb5b99fb1250e5f68/exceptiongroup-1.2.0-py3-none-any.whl (16 kB)\n",
      "Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ed/3c/cebfdcad015240014ff08b883d1c0c427f2ba45ae8c6572851b6ef136cad/marshmallow-3.20.1-py3-none-any.whl (49 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting typing-inspect<1,>=0.4.0 (from dataclasses-json<0.7,>=0.5.7->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/65/f3/107a22063bf27bdccf2024833d3445f4eea42b2e598abfbd46f6a63b6cb0/typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n",
      "Collecting jsonpointer>=1.9 (from jsonpatch<2.0,>=1.33->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/12/f6/0232cc0c617e195f06f810534d00b74d2f348fe71b2118009ad8ad31f878/jsonpointer-2.4-py2.py3-none-any.whl (7.8 kB)\n",
      "Collecting annotated-types>=0.4.0 (from pydantic<3,>=1->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/28/78/d31230046e58c207284c6b2c4e8d96e6d3cb4e52354721b944d3e1ee4aa5/annotated_types-0.6.0-py3-none-any.whl (12 kB)\n",
      "Collecting pydantic-core==2.14.6 (from pydantic<3,>=1->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/90/28/3c6843e6b203999be2660d3f114be196f2182dcac533dc764ad320c9184d/pydantic_core-2.14.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m69.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: typing-extensions>=4.6.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from pydantic<3,>=1->langchain==0.0.329) (4.8.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.329) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.329) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.329) (2023.11.17)\n",
      "Collecting greenlet!=0.4.17 (from SQLAlchemy<3,>=1.4->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/24/35/945d5b10648fec9b20bcc6df8952d20bb3bba76413cd71c1fdbee98f5616/greenlet-3.0.3-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (616 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m616.0/616.0 kB\u001b[0m \u001b[31m49.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: packaging>=17.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json<0.7,>=0.5.7->langchain==0.0.329) (23.2)\n",
      "Collecting mypy-extensions>=0.3.0 (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2a/e2/5d3f6ada4297caebe1a2add3b126fe800c96f56dbe5d1988a2cbe0b267aa/mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
      "Installing collected packages: tenacity, sniffio, pydantic-core, mypy-extensions, multidict, marshmallow, jsonpointer, greenlet, frozenlist, exceptiongroup, attrs, async-timeout, annotated-types, yarl, typing-inspect, SQLAlchemy, pydantic, jsonpatch, anyio, aiosignal, langsmith, dataclasses-json, aiohttp, langchain\n",
      "Successfully installed SQLAlchemy-2.0.23 aiohttp-3.9.1 aiosignal-1.3.1 annotated-types-0.6.0 anyio-3.7.1 async-timeout-4.0.3 attrs-23.1.0 dataclasses-json-0.6.3 exceptiongroup-1.2.0 frozenlist-1.4.1 greenlet-3.0.3 jsonpatch-1.33 jsonpointer-2.4 langchain-0.0.329 langsmith-0.0.75 marshmallow-3.20.1 multidict-6.0.4 mypy-extensions-1.0.0 pydantic-2.5.3 pydantic-core-2.14.6 sniffio-1.3.0 tenacity-8.2.3 typing-inspect-0.9.0 yarl-1.9.4\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting nltk==3.8.1\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a6/0a/0d20d2c0f16be91b9fa32a77b76c60f9baf6eba419e5ef5deca17af9c582/nltk-3.8.1-py3-none-any.whl (1.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m30.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting click (from nltk==3.8.1)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl (97 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.9/97.9 kB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting joblib (from nltk==3.8.1)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/10/40/d551139c85db202f1f384ba8bcf96aca2f329440a844f924c8a0040b6d02/joblib-1.3.2-py3-none-any.whl (302 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.2/302.2 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: regex>=2021.8.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk==3.8.1) (2023.5.5)\n",
      "Requirement already satisfied: tqdm in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk==3.8.1) (4.66.1)\n",
      "Installing collected packages: joblib, click, nltk\n",
      "Successfully installed click-8.1.7 joblib-1.3.2 nltk-3.8.1\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting unstructured==0.10.24\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/5d/10/f36ec76d07acee92ea7432c00e41d15ebf47df1db994778a1710da77e4a2/unstructured-0.10.24-py3-none-any.whl (1.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m24.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting chardet (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/38/6f/f5fbc992a329ee4e0f288c1fe0e2ad9485ed064cac731ed2fe47dcc38cbf/chardet-5.2.0-py3-none-any.whl (199 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m199.4/199.4 kB\u001b[0m \u001b[31m18.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting filetype (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/18/79/1b8fa1bb3568781e84c9200f951c735f3f157429f44be0495da55894d620/filetype-1.2.0-py2.py3-none-any.whl (19 kB)\n",
      "Collecting python-magic (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6c/73/9f872cb81fc5c3bb48f7227872c28975f998f3e7c2b1c16e95e6432bbb90/python_magic-0.4.27-py2.py3-none-any.whl (13 kB)\n",
      "Collecting lxml (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/18/02/5e28cbbfff53d4e227114f507cb03d02973f0ba0e5ea3e11aea66fcfe471/lxml-4.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (7.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.0/7.0 MB\u001b[0m \u001b[31m79.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m0:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: nltk in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from unstructured==0.10.24) (3.8.1)\n",
      "Collecting tabulate (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/40/44/4a5f08c96eb108af5cb50b41f76142f0afa346dfa99d5296fe7202a11854/tabulate-0.9.0-py3-none-any.whl (35 kB)\n",
      "Requirement already satisfied: requests in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from unstructured==0.10.24) (2.31.0)\n",
      "Collecting beautifulsoup4 (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/57/f4/a69c20ee4f660081a7dedb1ac57f29be9378e04edfcb90c526b923d4bebc/beautifulsoup4-4.12.2-py3-none-any.whl (142 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m143.0/143.0 kB\u001b[0m \u001b[31m18.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting emoji (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/03/40/91d0c9fe5a0b494c0fdbcacda4d203aea39f8293e69c70129389308ca928/emoji-2.9.0-py2.py3-none-any.whl (397 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m397.5/397.5 kB\u001b[0m \u001b[31m39.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: dataclasses-json in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from unstructured==0.10.24) (0.6.3)\n",
      "Collecting python-iso639 (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d4/3e/f20c6edf8b95db750a52269d3dffd6e6a4432989cc7731efef2a06c1d187/python_iso639-2023.12.11-py3-none-any.whl (275 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m275.1/275.1 kB\u001b[0m \u001b[31m31.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting langdetect (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/0e/72/a3add0e4eec4eb9e2569554f7c70f4a3c27712f40e3284d483e88094cc0e/langdetect-1.0.9.tar.gz (981 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m981.5/981.5 kB\u001b[0m \u001b[31m62.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: numpy in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from unstructured==0.10.24) (1.25.2)\n",
      "Collecting rapidfuzz (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/77/9b/c3bba754af8a0e095da8aad260b319e6949a035a3020d26366c695d2d9c3/rapidfuzz-3.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m80.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hCollecting backoff (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/df/73/b6e24bd22e6720ca8ee9a85a0c4a2971af8497d8f3193fa05390cbd46e09/backoff-2.2.1-py3-none-any.whl (15 kB)\n",
      "Collecting soupsieve>1.2 (from beautifulsoup4->unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/4c/f3/038b302fdfbe3be7da016777069f26ceefe11a681055ea1f7817546508e3/soupsieve-2.5-py3-none-any.whl (36 kB)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from dataclasses-json->unstructured==0.10.24) (3.20.1)\n",
      "Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from dataclasses-json->unstructured==0.10.24) (0.9.0)\n",
      "Requirement already satisfied: six in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langdetect->unstructured==0.10.24) (1.16.0)\n",
      "Requirement already satisfied: click in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->unstructured==0.10.24) (8.1.7)\n",
      "Requirement already satisfied: joblib in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->unstructured==0.10.24) (1.3.2)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->unstructured==0.10.24) (2023.5.5)\n",
      "Requirement already satisfied: tqdm in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->unstructured==0.10.24) (4.66.1)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->unstructured==0.10.24) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->unstructured==0.10.24) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->unstructured==0.10.24) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->unstructured==0.10.24) (2023.11.17)\n",
      "Requirement already satisfied: packaging>=17.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->unstructured==0.10.24) (23.2)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json->unstructured==0.10.24) (1.0.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json->unstructured==0.10.24) (4.8.0)\n",
      "Building wheels for collected packages: langdetect\n",
      "  Building wheel for langdetect (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for langdetect: filename=langdetect-1.0.9-py3-none-any.whl size=993224 sha256=afe213b6209dbbce0434b4eaeaa39e754fa1e0ff8b8a56908dfe8324c21f6c34\n",
      "  Stored in directory: /home/ma-user/.cache/pip/wheels/ee/e9/63/fe12d571f8675325c5e131236f64a52b7ed05da124bd628a74\n",
      "Successfully built langdetect\n",
      "Installing collected packages: filetype, tabulate, soupsieve, rapidfuzz, python-magic, python-iso639, lxml, langdetect, emoji, chardet, backoff, beautifulsoup4, unstructured\n",
      "Successfully installed backoff-2.2.1 beautifulsoup4-4.12.2 chardet-5.2.0 emoji-2.9.0 filetype-1.2.0 langdetect-1.0.9 lxml-4.9.4 python-iso639-2023.12.11 python-magic-0.4.27 rapidfuzz-3.6.0 soupsieve-2.5 tabulate-0.9.0 unstructured-0.10.24\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting sentence-transformers==2.2.2\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/20/9c/f07bd70d128fdb107bc02a0c702b9058b4fe147d0ba67b5a0f4c3cf15a54/sentence-transformers-2.2.2.tar.gz (85 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.0/86.0 kB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: transformers<5.0.0,>=4.6.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (4.30.2)\n",
      "Requirement already satisfied: tqdm in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (4.66.1)\n",
      "Requirement already satisfied: torch>=1.6.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (2.0.1)\n",
      "Collecting torchvision (from sentence-transformers==2.2.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/87/0f/88f023bf6176d9af0f85feedf4be129f9cf2748801c4d9c690739a10c100/torchvision-0.15.2-cp310-cp310-manylinux1_x86_64.whl (6.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.0/6.0 MB\u001b[0m \u001b[31m43.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: numpy in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (1.25.2)\n",
      "Collecting scikit-learn (from sentence-transformers==2.2.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/5c/e9/ee572691a3fb05555bcde41826faad29ae4bc1fb07982e7f53d54a176879/scikit_learn-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.8/10.8 MB\u001b[0m \u001b[31m58.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m0:01\u001b[0m\n",
      "\u001b[?25hCollecting scipy (from sentence-transformers==2.2.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/a8/cc/c36f3439f5d47c3b13833ce6687b43a040cc7638c502ac46b41e2d4f3d6f/scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m36.3/36.3 MB\u001b[0m \u001b[31m75.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: nltk in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (3.8.1)\n",
      "Requirement already satisfied: sentencepiece in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (0.1.99)\n",
      "Requirement already satisfied: huggingface-hub>=0.4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (0.16.4)\n",
      "Requirement already satisfied: filelock in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (3.12.2)\n",
      "Requirement already satisfied: fsspec in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2022.11.0)\n",
      "Requirement already satisfied: requests in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2.31.0)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (6.0.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (4.8.0)\n",
      "Requirement already satisfied: packaging>=20.9 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (23.2)\n",
      "Requirement already satisfied: sympy in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (1.12)\n",
      "Requirement already satisfied: networkx in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (2.8.8)\n",
      "Requirement already satisfied: jinja2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (3.1.2)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.7.101)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (8.5.0.96)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.10.3.66)\n",
      "Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (10.9.0.58)\n",
      "Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (10.2.10.91)\n",
      "Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.4.0.1)\n",
      "Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.7.4.91)\n",
      "Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (2.14.3)\n",
      "Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.7.91)\n",
      "Requirement already satisfied: triton==2.0.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (2.0.0)\n",
      "Requirement already satisfied: setuptools in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.6.0->sentence-transformers==2.2.2) (68.2.2)\n",
      "Requirement already satisfied: wheel in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.6.0->sentence-transformers==2.2.2) (0.41.2)\n",
      "Requirement already satisfied: cmake in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from triton==2.0.0->torch>=1.6.0->sentence-transformers==2.2.2) (3.27.2)\n",
      "Requirement already satisfied: lit in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from triton==2.0.0->torch>=1.6.0->sentence-transformers==2.2.2) (16.0.5)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2) (2023.5.5)\n",
      "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2) (0.13.3)\n",
      "Requirement already satisfied: safetensors>=0.3.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2) (0.3.3)\n",
      "Requirement already satisfied: click in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->sentence-transformers==2.2.2) (8.1.7)\n",
      "Requirement already satisfied: joblib in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->sentence-transformers==2.2.2) (1.3.2)\n",
      "Collecting threadpoolctl>=2.0.0 (from scikit-learn->sentence-transformers==2.2.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/61/cf/6e354304bcb9c6413c4e02a747b600061c21d38ba51e7e544ac7bc66aecc/threadpoolctl-3.1.0-py3-none-any.whl (14 kB)\n",
      "Collecting pillow!=8.3.*,>=5.3.0 (from torchvision->sentence-transformers==2.2.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/95/7b/71e2665760b5c33af00fa9bb6d6bca068b51bf021a4ceaeee03e18689f51/Pillow-10.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.5/3.5 MB\u001b[0m \u001b[31m66.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: MarkupSafe>=2.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from jinja2->torch>=1.6.0->sentence-transformers==2.2.2) (2.1.3)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2023.11.17)\n",
      "Requirement already satisfied: mpmath>=0.19 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sympy->torch>=1.6.0->sentence-transformers==2.2.2) (1.3.0)\n",
      "Building wheels for collected packages: sentence-transformers\n",
      "  Building wheel for sentence-transformers (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125923 sha256=63e18e388d12ffdbfb3094d106e043f9fac30f6a56e8ef04a80eb7b66bd84928\n",
      "  Stored in directory: /home/ma-user/.cache/pip/wheels/4f/9d/a5/9beabf87fdb3e143ae061b7bd2356c98c21bae0c908df108ee\n",
      "Successfully built sentence-transformers\n",
      "Installing collected packages: threadpoolctl, scipy, pillow, scikit-learn, torchvision, sentence-transformers\n",
      "Successfully installed pillow-10.1.0 scikit-learn-1.3.0 scipy-1.11.2 sentence-transformers-2.2.2 threadpoolctl-3.1.0 torchvision-0.15.2\n"
     ]
    }
   ],
   "source": [
    "!pip install transformers==4.30.2\n",
    "!pip install sentencepiece==0.1.99\n",
    "!pip install torch==2.0.1\n",
    "!pip install markdown==3.4.3\n",
    "!pip install faiss-gpu==1.7.2\n",
    "!pip install langchain==0.0.329 -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "!pip install nltk==3.8.1 -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "!pip install unstructured==0.10.24 -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "!pip install sentence-transformers==2.2.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6703d3e-8451-47e7-bac8-401cdb039be7",
   "metadata": {},
   "source": [
    "## 3. 中文文字匹配\n",
    "\n",
    "langchain支持很多中embedding模型，例如[text2vec-large-chinese](https://github.com/shibing624/text2vec)、[m3e-large](https://github.com/wangyingdong/m3e-base)、[bge-large-zh](https://github.com/jsonzhuwei/bge-large-zh)等，本文中使用text2vec-large-chinese模型实现。\n",
    "\n",
    "我们将待匹配的文字通过text2vec-large-chinese模型，转成嵌入向量，然后计算两个向量直接的相似性来进行匹配。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8eea0c09-399f-4934-9c37-dd994cefb3b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import nltk\n",
    "work_dir = '/home/ma-user/work'\n",
    "docs_path = os.path.join(work_dir, 'docs')\n",
    "nltk.data.path.append(os.path.join(work_dir, 'nltk_data'))\n",
    "\n",
    "import numpy as np\n",
    "from nltk import data\n",
    "from langchain.vectorstores import FAISS\n",
    "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
    "from langchain.text_splitter import CharacterTextSplitter,MarkdownTextSplitter\n",
    "from langchain.document_loaders import UnstructuredFileLoader,UnstructuredMarkdownLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6cd3b3e0-4795-4573-a188-1d8acb9c16ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_model = 'text2vec-large-chinese'\n",
    "\n",
    "#基于余弦相似性公式计算两个向量之间的相似度\n",
    "def get_cos_similar(v1: list, v2: list):\n",
    "    num = float(np.dot(v1, v2))  # 向量点乘\n",
    "    denom = np.linalg.norm(v1) * np.linalg.norm(v2)  # 求模长的乘积\n",
    "    return 0.5 + 0.5 * (num / denom) if denom != 0 else 0\n",
    "\n",
    "#加载text2vec-large-chinese模型\n",
    "def load_embeddings():\n",
    "    embedding_model_path = os.path.join(work_dir, embedding_model)\n",
    "    embeddings = HuggingFaceEmbeddings(model_name=embedding_model_path)\n",
    "    return embeddings\n",
    "\n",
    "#计算两段文字的相似度\n",
    "def get_embedding_sim(s1, s2, embeddings):\n",
    "    embedding1 = embeddings.embed_query(s1)#将文本转为向量\n",
    "    print('embedding1: ', len(embedding1))\n",
    "    embedding2 = embeddings.embed_query(s2)\n",
    "    sim = get_cos_similar(embedding1, embedding2)\n",
    "    print('sim of \\'{0}\\' and \\'{1}\\' is : {2}'.format(s1, s2, sim))\n",
    "    return sim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2b533d4c-fbe3-4224-97fc-d36fed4478e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "No sentence-transformers model found with name /home/ma-user/work/text2vec-large-chinese. Creating a new one with MEAN pooling.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "embedding1:  1024\n",
      "sim of '我今天心情很差' and '我今天很不开心' is : 0.8813112714604454\n",
      "embedding1:  1024\n",
      "sim of '我今天心情很差' and 'what are you弄啥嘞' is : 0.5937208396102376\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.5937208396102376"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence1 = \"我今天心情很差\"\n",
    "sentence2 = \"我今天很不开心\"\n",
    "sentence3 = \"what are you弄啥嘞\"\n",
    "embeddings = load_embeddings()\n",
    "get_embedding_sim(sentence1, sentence2, embeddings)\n",
    "get_embedding_sim(sentence1, sentence3, embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2760277-0f61-4979-beef-44d7d7041311",
   "metadata": {},
   "source": [
    "## 4. 文档查询\n",
    "对于文档查询，我们首先也是将分割后的文档转成嵌入向量，然后存储到向量数据库，再根据查询条件，从向量数据库进行搜索。\n",
    "\n",
    "langchain支持的向量数据库有很多种，例如：[FAISS](https://github.com/facebookresearch/faiss)、[Milvus](https://github.com/milvus-io/milvus)、[PGVector](https://github.com/pgvector/pgvector)等，本文使用的是FAISS。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "931ca856-6c99-4ace-935e-3d80f6264f55",
   "metadata": {},
   "outputs": [],
   "source": [
    "#加载txt文件\n",
    "def load_txt_file(txt_file):    \n",
    "    loader = UnstructuredFileLoader(os.path.join(work_dir, txt_file))\n",
    "    docs = loader.load()\n",
    "    return docs\n",
    "\n",
    "#加载md文件\n",
    "def load_md_file(md_file):    \n",
    "    loader = UnstructuredMarkdownLoader(os.path.join(work_dir, md_file))\n",
    "    docs = loader.load()\n",
    "    return docs\n",
    "\n",
    "#分割txt文件\n",
    "def load_txt_splitter(txt_file, chunk_size=100, chunk_overlap=20):\n",
    "    docs = load_txt_file(txt_file)\n",
    "    text_splitter = CharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)\n",
    "    split_docs = text_splitter.split_documents(docs)\n",
    "    return split_docs\n",
    "\n",
    "#分割md文件\n",
    "def load_md_splitter(md_file, chunk_size=100, chunk_overlap=20):\n",
    "    docs = load_md_file(md_file)\n",
    "    text_splitter = MarkdownTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)\n",
    "    split_docs = text_splitter.split_documents(docs)\n",
    "    return split_docs\n",
    "\n",
    "#分割docs_path目录下的文件，并将其转为向量，放到FAISS向量数据库中\n",
    "def load_vector_store(docs_path):\n",
    "    split_docs = []\n",
    "    for doc in os.listdir(docs_path):\n",
    "        doc_path = f'{docs_path}/{doc}'\n",
    "        if doc_path.endswith('.txt'):\n",
    "            docs = load_txt_splitter(doc_path)\n",
    "            split_docs.extend(docs)\n",
    "        elif doc_path.endswith('.md'):\n",
    "            docs = load_md_splitter(doc_path)\n",
    "            split_docs.extend(docs)\n",
    "        else:\n",
    "            print('不支持的文件类型:', doc_path)\n",
    "            continue\n",
    "    embeddings = load_embeddings()\n",
    "    vector_store = FAISS.from_documents(split_docs, embeddings)\n",
    "    return vector_store\n",
    "\n",
    "#从向量数据集进行内容查询\n",
    "def sim_search(query, vector_store):\n",
    "    #similarity_search_with_score返回相似的文档内容和查询与文档的距离分数\n",
    "    #返回的距离分数是L2距离。因此，得分越低越好。\n",
    "    re = vector_store.similarity_search_with_score(query)\n",
    "    print('query result: ', re)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e5e3ddf3-f704-433b-9c39-e8dab617fd2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "不支持的文件类型: /home/ma-user/work/docs/.ipynb_checkpoints\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 146, which is longer than the specified 100\n",
      "No sentence-transformers model found with name /home/ma-user/work/text2vec-large-chinese. Creating a new one with MEAN pooling.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "不支持的文件类型: /home/ma-user/work/docs/modelbox.jpg\n",
      "不支持的文件类型: /home/ma-user/work/docs/modelbox.pdf\n",
      "query result:  [(Document(page_content='ModelBox支持两种方式运行，一种是服务化，一种是SDK，开发者可以按照下表选择相关的开发模式。', metadata={'source': '/home/ma-user/work/docs/modelbox.txt'}), 420.72437), (Document(page_content='2. SDK：ModelBox提供了ModelBox开发库，使用于扩展现有应用支持高性能AI推理，专注AI推理业务，支持c++，Python集成', metadata={'source': '/home/ma-user/work/docs/modelbox.txt'}), 587.23193), (Document(page_content='如果是第一次创建工程，在ModelBox', metadata={'source': '/home/ma-user/work/docs/第一个应用.md'}), 598.4208), (Document(page_content='也就是说，ModelBox的Pipeline模式，首先需要将应用的流程图构建出来，再分别实现图中的每个模块（ModelBox中称为“功能单元”），对于上面的视频应用：读取摄像头并输出原始画面，对应的', metadata={'source': '/home/ma-user/work/docs/第一个应用.md'}), 646.28265)]\n"
     ]
    }
   ],
   "source": [
    "query = \"ModelBox支持哪两种方式\"\n",
    "vector_store = load_vector_store(docs_path)\n",
    "sim_search(query, vector_store)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45231615-a4a9-48ae-a97d-d8932d6f9eb1",
   "metadata": {},
   "source": [
    "### 创新性的案例\n",
    "#### 完成课程实践的基础上，重新自己起一个Notebook，有创新点\n",
    "#### 基于LangChain+ChatGLM3的本地知识库问答"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a511bca6-7538-4a1f-8a3a-3dbd607e821d",
   "metadata": {},
   "source": [
    "#### 2.1 下载ChatGLM3模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "068311b1-ebef-4ad2-97b0-ecda1375a407",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Using MoXing-v2.1.0.5d9c87c8-5d9c87c8\n",
      "INFO:root:Using OBS-Python-SDK-3.20.9.1\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import moxing as mox\n",
    "\n",
    "#拷贝chatglm3模型到chatglm3-6b目录下\n",
    "work_dir = '/home/ma-user/work'\n",
    "obs_path = 'obs://dtse-models/tar-models/chatglm3-6b.tar'\n",
    "ma_path = os.path.join(work_dir, 'chatglm3-6b.tar')\n",
    "mox.file.copy(obs_path, ma_path)\n",
    "\n",
    "mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/case_zoo/langchain-ChatGLM/file/service','service' )\n",
    "mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/course/ModelBox/frpc_linux_amd64', '/home/ma-user/work/frpc_linux_amd64')\n",
    "# mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/course/ModelBox/frpc_linux_amd64', '/home/ma-user/work/frpc_linux_amd64')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f5873e7-1222-4a5b-8ae9-45f16ab41493",
   "metadata": {},
   "source": [
    "进入chatglm3目录，解压模型压缩包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b1e17b1d-b806-4087-bd85-e4f174db60d8",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ma-user/work\n",
      "chatglm3-6b/\n",
      "chatglm3-6b/.gitattributes\n",
      "chatglm3-6b/MODEL_LICENSE\n",
      "chatglm3-6b/README.md\n",
      "chatglm3-6b/config.json\n",
      "chatglm3-6b/configuration_chatglm.py\n",
      "chatglm3-6b/modeling_chatglm.py\n",
      "chatglm3-6b/pytorch_model-00001-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00002-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00003-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00004-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00005-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00006-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00007-of-00007.bin\n",
      "chatglm3-6b/pytorch_model.bin.index.json\n",
      "chatglm3-6b/quantization.py\n",
      "chatglm3-6b/tokenization_chatglm.py\n",
      "chatglm3-6b/tokenizer.model\n",
      "chatglm3-6b/tokenizer_config.json\n"
     ]
    }
   ],
   "source": [
    "os.chdir(work_dir)\n",
    "!pwd\n",
    "!tar -xvf chatglm3-6b.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7893f635-768c-43d7-bf72-0af90d3ef86b",
   "metadata": {},
   "source": [
    "下载nltk_data数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e6368a1c-daf2-4097-b6cb-528d5043264c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import moxing as mox\n",
    "\n",
    "work_dir = '/home/ma-user/work'\n",
    "obs_path = 'obs://dtse-models/tar-models/nltk_data.tar'\n",
    "ma_path = os.path.join(work_dir, 'nltk_data.tar')\n",
    "mox.file.copy(obs_path, ma_path)\n",
    "\n",
    "mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/case_zoo/langchain-ChatGLM/file/docs','/home/ma-user/work/docs')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da828f6c-eba6-4659-a244-598188a93486",
   "metadata": {},
   "source": [
    "进入nltk_data目录，解压数据压缩包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c0e0bc2a-dfc4-4240-a078-fded495e6574",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ma-user/work\n",
      "nltk_data/\n",
      "nltk_data/misc/\n",
      "nltk_data/misc/mwa_ppdb.zip\n",
      "nltk_data/misc/perluniprops.xml\n",
      "nltk_data/misc/mwa_ppdb.xml\n",
      "nltk_data/misc/perluniprops.zip\n",
      "nltk_data/tokenizers/\n",
      "nltk_data/tokenizers/punkt/\n",
      "nltk_data/tokenizers/punkt/french.pickle\n",
      "nltk_data/tokenizers/punkt/polish.pickle\n",
      "nltk_data/tokenizers/punkt/.DS_Store\n",
      "nltk_data/tokenizers/punkt/portuguese.pickle\n",
      "nltk_data/tokenizers/punkt/german.pickle\n",
      "nltk_data/tokenizers/punkt/swedish.pickle\n",
      "nltk_data/tokenizers/punkt/malayalam.pickle\n",
      "nltk_data/tokenizers/punkt/estonian.pickle\n",
      "nltk_data/tokenizers/punkt/finnish.pickle\n",
      "nltk_data/tokenizers/punkt/spanish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/\n",
      "nltk_data/tokenizers/punkt/PY3/french.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/polish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/portuguese.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/german.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/swedish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/malayalam.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/estonian.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/finnish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/spanish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/danish.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/greek.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/russian.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/slovene.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/norwegian.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/english.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/dutch.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/italian.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/czech.pickle\n",
      "nltk_data/tokenizers/punkt/PY3/README\n",
      "nltk_data/tokenizers/punkt/PY3/turkish.pickle\n",
      "nltk_data/tokenizers/punkt/danish.pickle\n",
      "nltk_data/tokenizers/punkt/greek.pickle\n",
      "nltk_data/tokenizers/punkt/russian.pickle\n",
      "nltk_data/tokenizers/punkt/slovene.pickle\n",
      "nltk_data/tokenizers/punkt/norwegian.pickle\n",
      "nltk_data/tokenizers/punkt/english.pickle\n",
      "nltk_data/tokenizers/punkt/dutch.pickle\n",
      "nltk_data/tokenizers/punkt/italian.pickle\n",
      "nltk_data/tokenizers/punkt/czech.pickle\n",
      "nltk_data/tokenizers/punkt/README\n",
      "nltk_data/tokenizers/punkt/turkish.pickle\n",
      "nltk_data/tokenizers/punkt.zip\n",
      "nltk_data/tokenizers/punkt.xml\n",
      "nltk_data/sentiment/\n",
      "nltk_data/sentiment/vader_lexicon.xml\n",
      "nltk_data/sentiment/vader_lexicon.zip\n",
      "nltk_data/chunkers/\n",
      "nltk_data/chunkers/maxent_ne_chunker.xml\n",
      "nltk_data/chunkers/maxent_ne_chunker.zip\n",
      "nltk_data/help/\n",
      "nltk_data/help/tagsets.xml\n",
      "nltk_data/help/tagsets.zip\n",
      "nltk_data/taggers/\n",
      "nltk_data/taggers/averaged_perceptron_tagger.xml\n",
      "nltk_data/taggers/averaged_perceptron_tagger.zip\n",
      "nltk_data/taggers/universal_tagset.xml\n",
      "nltk_data/taggers/averaged_perceptron_tagger_ru.zip\n",
      "nltk_data/taggers/averaged_perceptron_tagger/\n",
      "nltk_data/taggers/averaged_perceptron_tagger/averaged_perceptron_tagger.pickle\n",
      "nltk_data/taggers/maxent_treebank_pos_tagger.zip\n",
      "nltk_data/taggers/universal_tagset.zip\n",
      "nltk_data/taggers/averaged_perceptron_tagger_ru.xml\n",
      "nltk_data/taggers/maxent_treebank_pos_tagger.xml\n",
      "nltk_data/corpora/\n",
      "nltk_data/corpora/cess_cat.xml\n",
      "nltk_data/corpora/twitter_samples.xml\n",
      "nltk_data/corpora/udhr.zip\n",
      "nltk_data/corpora/ieer.zip\n",
      "nltk_data/corpora/senseval.xml\n",
      "nltk_data/corpora/nonbreaking_prefixes.zip\n",
      "nltk_data/corpora/mac_morpho.xml\n",
      "nltk_data/corpora/mte_teip5.xml\n",
      "nltk_data/corpora/omw.zip\n",
      "nltk_data/corpora/reuters.xml\n",
      "nltk_data/corpora/shakespeare.zip\n",
      "nltk_data/corpora/propbank.zip\n",
      "nltk_data/corpora/rte.zip\n",
      "nltk_data/corpora/ptb3.zip\n",
      "nltk_data/corpora/kimmo.xml\n",
      "nltk_data/corpora/names.xml\n",
      "nltk_data/corpora/comparative_sentences.zip\n",
      "nltk_data/corpora/ycoe.zip\n",
      "nltk_data/corpora/nps_chat.zip\n",
      "nltk_data/corpora/sentiwordnet.zip\n",
      "nltk_data/corpora/smultron.zip\n",
      "nltk_data/corpora/lin_thesaurus.zip\n",
      "nltk_data/corpora/jeita.zip\n",
      "nltk_data/corpora/chat80.xml\n",
      "nltk_data/corpora/knbc.zip\n",
      "nltk_data/corpora/sentence_polarity.xml\n",
      "nltk_data/corpora/conll2007.zip\n",
      "nltk_data/corpora/panlex_swadesh.xml\n",
      "nltk_data/corpora/verbnet3.zip\n",
      "nltk_data/corpora/wordnet2022.zip\n",
      "nltk_data/corpora/nombank.1.0.xml\n",
      "nltk_data/corpora/comtrans.xml\n",
      "nltk_data/corpora/stopwords.xml\n",
      "nltk_data/corpora/verbnet3.xml\n",
      "nltk_data/corpora/conll2002.xml\n",
      "nltk_data/corpora/omw-1.4.zip\n",
      "nltk_data/corpora/qc.zip\n",
      "nltk_data/corpora/ppattach.xml\n",
      "nltk_data/corpora/machado.xml\n",
      "nltk_data/corpora/wordnet2021.xml\n",
      "nltk_data/corpora/stopwords.zip\n",
      "nltk_data/corpora/framenet_v15.xml\n",
      "nltk_data/corpora/ptb.zip\n",
      "nltk_data/corpora/lin_thesaurus.xml\n",
      "nltk_data/corpora/product_reviews_1.zip\n",
      "nltk_data/corpora/cess_esp.zip\n",
      "nltk_data/corpora/words.xml\n",
      "nltk_data/corpora/nps_chat.xml\n",
      "nltk_data/corpora/reuters.zip\n",
      "nltk_data/corpora/brown_tei.xml\n",
      "nltk_data/corpora/toolbox.zip\n",
      "nltk_data/corpora/swadesh.xml\n",
      "nltk_data/corpora/ppattach.zip\n",
      "nltk_data/corpora/opinion_lexicon.zip\n",
      "nltk_data/corpora/semcor.zip\n",
      "nltk_data/corpora/cmudict.xml\n",
      "nltk_data/corpora/machado.zip\n",
      "nltk_data/corpora/subjectivity.zip\n",
      "nltk_data/corpora/pl196x.zip\n",
      "nltk_data/corpora/city_database.xml\n",
      "nltk_data/corpora/treebank.zip\n",
      "nltk_data/corpora/pil.zip\n",
      "nltk_data/corpora/mac_morpho.zip\n",
      "nltk_data/corpora/masc_tagged.xml\n",
      "nltk_data/corpora/universal_treebanks_v20.zip\n",
      "nltk_data/corpora/genesis.xml\n",
      "nltk_data/corpora/indian.xml\n",
      "nltk_data/corpora/timit.xml\n",
      "nltk_data/corpora/state_union.zip\n",
      "nltk_data/corpora/framenet_v17.xml\n",
      "nltk_data/corpora/opinion_lexicon.xml\n",
      "nltk_data/corpora/problem_reports.xml\n",
      "nltk_data/corpora/wordnet.zip\n",
      "nltk_data/corpora/framenet_v17.zip\n",
      "nltk_data/corpora/subjectivity.xml\n",
      "nltk_data/corpora/sentence_polarity.zip\n",
      "nltk_data/corpora/semcor.xml\n",
      "nltk_data/corpora/city_database.zip\n",
      "nltk_data/corpora/unicode_samples.xml\n",
      "nltk_data/corpora/bcp47.xml\n",
      "nltk_data/corpora/dolch.zip\n",
      "nltk_data/corpora/ptb.xml\n",
      "nltk_data/corpora/switchboard.xml\n",
      "nltk_data/corpora/wordnet2021.zip\n",
      "nltk_data/corpora/extended_omw.zip\n",
      "nltk_data/corpora/framenet_v15.zip\n",
      "nltk_data/corpora/genesis.zip\n",
      "nltk_data/corpora/movie_reviews.zip\n",
      "nltk_data/corpora/abc.zip\n",
      "nltk_data/corpora/dependency_treebank.zip\n",
      "nltk_data/corpora/unicode_samples.zip\n",
      "nltk_data/corpora/comtrans.zip\n",
      "nltk_data/corpora/nonbreaking_prefixes.xml\n",
      "nltk_data/corpora/pl196x.xml\n",
      "nltk_data/corpora/bcp47.zip\n",
      "nltk_data/corpora/switchboard.zip\n",
      "nltk_data/corpora/pros_cons.zip\n",
      "nltk_data/corpora/wordnet_ic.zip\n",
      "nltk_data/corpora/udhr.xml\n",
      "nltk_data/corpora/dolch.xml\n",
      "nltk_data/corpora/shakespeare.xml\n",
      "nltk_data/corpora/verbnet.zip\n",
      "nltk_data/corpora/pe08.zip\n",
      "nltk_data/corpora/conll2007.xml\n",
      "nltk_data/corpora/brown.xml\n",
      "nltk_data/corpora/propbank.xml\n",
      "nltk_data/corpora/biocreative_ppi.zip\n",
      "nltk_data/corpora/product_reviews_2.xml\n",
      "nltk_data/corpora/senseval.zip\n",
      "nltk_data/corpora/unicode.notes\n",
      "nltk_data/corpora/gutenberg.zip\n",
      "nltk_data/corpora/sinica_treebank.zip\n",
      "nltk_data/corpora/state_union.xml\n",
      "nltk_data/corpora/product_reviews_2.zip\n",
      "nltk_data/corpora/ieer.xml\n",
      "nltk_data/corpora/paradigms.xml\n",
      "nltk_data/corpora/wordnet31.zip\n",
      "nltk_data/corpora/cess_esp.xml\n",
      "nltk_data/corpora/indian.zip\n",
      "nltk_data/corpora/conll2000.xml\n",
      "nltk_data/corpora/abc.xml\n",
      "nltk_data/corpora/timit.zip\n",
      "nltk_data/corpora/udhr2.xml\n",
      "nltk_data/corpora/floresta.xml\n",
      "nltk_data/corpora/inaugural.zip\n",
      "nltk_data/corpora/panlex_swadesh.zip\n",
      "nltk_data/corpora/gazetteers.xml\n",
      "nltk_data/corpora/product_reviews_1.xml\n",
      "nltk_data/corpora/movie_reviews.xml\n",
      "nltk_data/corpora/biocreative_ppi.xml\n",
      "nltk_data/corpora/webtext.xml\n",
      "nltk_data/corpora/udhr2.zip\n",
      "nltk_data/corpora/twitter_samples.zip\n",
      "nltk_data/corpora/comparative_sentences.xml\n",
      "nltk_data/corpora/knbc.xml\n",
      "nltk_data/corpora/cmudict.zip\n",
      "nltk_data/corpora/omw.xml\n",
      "nltk_data/corpora/masc_tagged.zip\n",
      "nltk_data/corpora/problem_reports.zip\n",
      "nltk_data/corpora/cess_cat.zip\n",
      "nltk_data/corpora/conll2000.zip\n",
      "nltk_data/corpora/swadesh.zip\n",
      "nltk_data/corpora/pil.xml\n",
      "nltk_data/corpora/verbnet.xml\n",
      "nltk_data/corpora/sinica_treebank.xml\n",
      "nltk_data/corpora/kimmo.zip\n",
      "nltk_data/corpora/conll2002.zip\n",
      "nltk_data/corpora/floresta.zip\n",
      "nltk_data/corpora/treebank.xml\n",
      "nltk_data/corpora/listing.csv\n",
      "nltk_data/corpora/nombank.1.0.zip\n",
      "nltk_data/corpora/listing.csv.zip\n",
      "nltk_data/corpora/words.zip\n",
      "nltk_data/corpora/webtext.zip\n",
      "nltk_data/corpora/crubadan.xml\n",
      "nltk_data/corpora/pros_cons.xml\n",
      "nltk_data/corpora/omw-1.4.xml\n",
      "nltk_data/corpora/brown_tei.zip\n",
      "nltk_data/corpora/jeita.xml\n",
      "nltk_data/corpora/ycoe.xml\n",
      "nltk_data/corpora/alpino.zip\n",
      "nltk_data/corpora/universal_treebanks_v20.xml\n",
      "nltk_data/corpora/gazetteers.zip\n",
      "nltk_data/corpora/europarl_raw.xml\n",
      "nltk_data/corpora/paradigms.zip\n",
      "nltk_data/corpora/qc.xml\n",
      "nltk_data/corpora/chat80.zip\n",
      "nltk_data/corpora/europarl_raw.zip\n",
      "nltk_data/corpora/mte_teip5.zip\n",
      "nltk_data/corpora/toolbox.xml\n",
      "nltk_data/corpora/brown.zip\n",
      "nltk_data/corpora/sentiwordnet.xml\n",
      "nltk_data/corpora/wordnet_ic.xml\n",
      "nltk_data/corpora/gutenberg.xml\n",
      "nltk_data/corpora/names.zip\n",
      "nltk_data/corpora/extended_omw.xml\n",
      "nltk_data/corpora/wordnet.xml\n",
      "nltk_data/corpora/inaugural.xml\n",
      "nltk_data/corpora/wordnet31.xml\n",
      "nltk_data/corpora/crubadan.zip\n",
      "nltk_data/corpora/wordnet2022.xml\n",
      "nltk_data/corpora/alpino.xml\n",
      "nltk_data/corpora/smultron.xml\n",
      "nltk_data/corpora/pe08.xml\n",
      "nltk_data/corpora/rte.xml\n",
      "nltk_data/corpora/dependency_treebank.xml\n",
      "nltk_data/stemmers/\n",
      "nltk_data/stemmers/snowball_data.xml\n",
      "nltk_data/stemmers/rslp.xml\n",
      "nltk_data/stemmers/snowball_data.zip\n",
      "nltk_data/stemmers/porter_test.xml\n",
      "nltk_data/stemmers/porter_test.zip\n",
      "nltk_data/stemmers/rslp.zip\n",
      "nltk_data/grammars/\n",
      "nltk_data/grammars/book_grammars.zip\n",
      "nltk_data/grammars/book_grammars.xml\n",
      "nltk_data/grammars/spanish_grammars.xml\n",
      "nltk_data/grammars/sample_grammars.xml\n",
      "nltk_data/grammars/large_grammars.xml\n",
      "nltk_data/grammars/large_grammars.zip\n",
      "nltk_data/grammars/basque_grammars.xml\n",
      "nltk_data/grammars/spanish_grammars.zip\n",
      "nltk_data/grammars/sample_grammars.zip\n",
      "nltk_data/grammars/basque_grammars.zip\n",
      "nltk_data/models/\n",
      "nltk_data/models/word2vec_sample.xml\n",
      "nltk_data/models/moses_sample.zip\n",
      "nltk_data/models/moses_sample.xml\n",
      "nltk_data/models/bllip_wsj_no_aux.xml\n",
      "nltk_data/models/word2vec_sample.zip\n",
      "nltk_data/models/bllip_wsj_no_aux.zip\n",
      "nltk_data/models/wmt15_eval.zip\n",
      "nltk_data/models/wmt15_eval.xml\n"
     ]
    }
   ],
   "source": [
    "os.chdir(work_dir)\n",
    "!pwd\n",
    "!tar -xvf nltk_data.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93f1fea7-a7a9-4f82-9d89-132e25b04fb4",
   "metadata": {},
   "source": [
    "下载text2vec-large-chinese模型，用于中文通用语义匹配"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d53aef65-b6db-44f8-8dd1-ffc643bf5bd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import moxing as mox\n",
    "\n",
    "work_dir = '/home/ma-user/work'\n",
    "\n",
    "obs_path = 'obs://dtse-models/tar-models/text2vec-large-chinese.tar'\n",
    "ma_path = os.path.join(work_dir, 'text2vec-large-chinese.tar')\n",
    "mox.file.copy(obs_path, ma_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "358c53b6-d807-4e71-a70e-107866e01ab0",
   "metadata": {},
   "source": [
    "进入text2vec-large-chinese目录，解压模型压缩包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "118426b1-6b28-4b68-a7ae-7d0eb69c3ec4",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ma-user/work\n",
      "text2vec-large-chinese/\n",
      "text2vec-large-chinese/.gitattributes\n",
      "text2vec-large-chinese/README.md\n",
      "text2vec-large-chinese/config.json\n",
      "text2vec-large-chinese/eval_results.txt\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/blobs/\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/blobs/eaf5cb71c0eeab7db3c5171da504e5867b3f67a78e07bdba9b52d334ae35adb3.lock\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/refs/\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/refs/main\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/snapshots/\n",
      "text2vec-large-chinese/models--GanymedeNil--text2vec-large-chinese/snapshots/064717f2acfd7253bea91079d59b82e50b58c886/\n",
      "text2vec-large-chinese/pytorch_model.bin\n",
      "text2vec-large-chinese/special_tokens_map.json\n",
      "text2vec-large-chinese/tmpqlu9nxcm\n",
      "text2vec-large-chinese/tokenizer.json\n",
      "text2vec-large-chinese/tokenizer_config.json\n",
      "text2vec-large-chinese/vocab.txt\n"
     ]
    }
   ],
   "source": [
    "os.chdir(work_dir)\n",
    "!pwd\n",
    "!tar -xvf text2vec-large-chinese.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fab27421-1273-48d7-8999-b4d6d53e9a3a",
   "metadata": {},
   "source": [
    "### 2.2 配置环境\n",
    "\n",
    "本案例依赖Python3.10.10及以上环境，因此我们首先创建虚拟环境："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "825f23cd-b746-4ce1-87e5-229dd6212182",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!\n",
      "  RequestsDependencyWarning)\n",
      "Collecting package metadata (current_repodata.json): done\n",
      "Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.\n",
      "Collecting package metadata (repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/ma-user/anaconda3/envs/python-3.10.10\n",
      "\n",
      "  added / updated specs:\n",
      "    - python=3.10.10\n",
      "\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    _libgcc_mutex-0.1          |             main           3 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    _openmp_mutex-5.1          |            1_gnu          21 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    bzip2-1.0.8                |       h7b6447c_0          78 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    ca-certificates-2023.12.12 |       h06a4308_0         126 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    ld_impl_linux-64-2.38      |       h1181459_1         654 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    libffi-3.4.4               |       h6a678d5_0         142 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    libgcc-ng-11.2.0           |       h1234567_1         5.3 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    libgomp-11.2.0             |       h1234567_1         474 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    libuuid-1.41.5             |       h5eee18b_0          27 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    ncurses-6.4                |       h6a678d5_0         914 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    openssl-1.1.1w             |       h7f8727e_0         3.7 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    pip-23.3.1                 |  py310h06a4308_0         2.7 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    python-3.10.10             |       h7a1cb2a_2        26.9 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    readline-8.2               |       h5eee18b_0         357 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    setuptools-68.2.2          |  py310h06a4308_0         957 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    sqlite-3.41.2              |       h5eee18b_0         1.2 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    tk-8.6.12                  |       h1ccaba5_0         3.0 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    tzdata-2023c               |       h04d1e81_0         116 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    wheel-0.41.2               |  py310h06a4308_0         109 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    xz-5.4.5                   |       h5eee18b_0         646 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    zlib-1.2.13                |       h5eee18b_0         103 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:        52.2 MB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  _libgcc_mutex      anaconda/pkgs/main/linux-64::_libgcc_mutex-0.1-main\n",
      "  _openmp_mutex      anaconda/pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu\n",
      "  bzip2              anaconda/pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0\n",
      "  ca-certificates    anaconda/pkgs/main/linux-64::ca-certificates-2023.12.12-h06a4308_0\n",
      "  ld_impl_linux-64   anaconda/pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1\n",
      "  libffi             anaconda/pkgs/main/linux-64::libffi-3.4.4-h6a678d5_0\n",
      "  libgcc-ng          anaconda/pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1\n",
      "  libgomp            anaconda/pkgs/main/linux-64::libgomp-11.2.0-h1234567_1\n",
      "  libstdcxx-ng       anaconda/pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1\n",
      "  libuuid            anaconda/pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0\n",
      "  ncurses            anaconda/pkgs/main/linux-64::ncurses-6.4-h6a678d5_0\n",
      "  openssl            anaconda/pkgs/main/linux-64::openssl-1.1.1w-h7f8727e_0\n",
      "  pip                anaconda/pkgs/main/linux-64::pip-23.3.1-py310h06a4308_0\n",
      "  python             anaconda/pkgs/main/linux-64::python-3.10.10-h7a1cb2a_2\n",
      "  readline           anaconda/pkgs/main/linux-64::readline-8.2-h5eee18b_0\n",
      "  setuptools         anaconda/pkgs/main/linux-64::setuptools-68.2.2-py310h06a4308_0\n",
      "  sqlite             anaconda/pkgs/main/linux-64::sqlite-3.41.2-h5eee18b_0\n",
      "  tk                 anaconda/pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0\n",
      "  tzdata             anaconda/pkgs/main/noarch::tzdata-2023c-h04d1e81_0\n",
      "  wheel              anaconda/pkgs/main/linux-64::wheel-0.41.2-py310h06a4308_0\n",
      "  xz                 anaconda/pkgs/main/linux-64::xz-5.4.5-h5eee18b_0\n",
      "  zlib               anaconda/pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0\n",
      "\n",
      "\n",
      "\n",
      "Downloading and Extracting Packages\n",
      "_libgcc_mutex-0.1    | 3 KB      | ##################################### | 100% \n",
      "zlib-1.2.13          | 103 KB    | ##################################### | 100% \n",
      "bzip2-1.0.8          | 78 KB     | ##################################### | 100% \n",
      "openssl-1.1.1w       | 3.7 MB    | ##################################### | 100% \n",
      "ncurses-6.4          | 914 KB    | ##################################### | 100% \n",
      "ca-certificates-2023 | 126 KB    | ##################################### | 100% \n",
      "libstdcxx-ng-11.2.0  | 4.7 MB    | ##################################### | 100% \n",
      "ld_impl_linux-64-2.3 | 654 KB    | ##################################### | 100% \n",
      "wheel-0.41.2         | 109 KB    | ##################################### | 100% \n",
      "sqlite-3.41.2        | 1.2 MB    | ##################################### | 100% \n",
      "libgcc-ng-11.2.0     | 5.3 MB    | ##################################### | 100% \n",
      "xz-5.4.5             | 646 KB    | ##################################### | 100% \n",
      "_openmp_mutex-5.1    | 21 KB     | ##################################### | 100% \n",
      "python-3.10.10       | 26.9 MB   | ##################################### | 100% \n",
      "setuptools-68.2.2    | 957 KB    | ##################################### | 100% \n",
      "tk-8.6.12            | 3.0 MB    | ##################################### | 100% \n",
      "readline-8.2         | 357 KB    | ##################################### | 100% \n",
      "libuuid-1.41.5       | 27 KB     | ##################################### | 100% \n",
      "pip-23.3.1           | 2.7 MB    | ##################################### | 100% \n",
      "libffi-3.4.4         | 142 KB    | ##################################### | 100% \n",
      "libgomp-11.2.0       | 474 KB    | ##################################### | 100% \n",
      "tzdata-2023c         | 116 KB    | ##################################### | 100% \n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "#\n",
      "# To activate this environment, use\n",
      "#\n",
      "#     $ conda activate python-3.10.10\n",
      "#\n",
      "# To deactivate an active environment, use\n",
      "#\n",
      "#     $ conda deactivate\n",
      "\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting ipykernel\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/5d/4b/ffb537e392e730c9a5b02758f9c87077d9087bcb0d957853e13f121e5ea7/ipykernel-6.23.1-py3-none-any.whl (152 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.2/152.2 kB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting comm>=0.1.1 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/74/f3/b88d7e1dadf741550c56b70d7ce62673354fddb68e143d193ceb80224208/comm-0.1.3-py3-none-any.whl (6.6 kB)\n",
      "Collecting debugpy>=1.6.5 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/1f/19/345c21f6b62acf556c39e4358a22b0ad868fecb462c1041c13513d229b33/debugpy-1.6.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m29.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting ipython>=7.23.1 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/09/db/d641ee07f319002393524b6c5a8b47370520dcb2b6166a0972cfe9398c60/ipython-8.10.0-py3-none-any.whl (784 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m784.3/784.3 kB\u001b[0m \u001b[31m50.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting jupyter-client>=6.1.12 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/07/37/4019d2c41ca333c08dfdfeb84c0fc0368c8defbbd3c8f0c9a530851e5813/jupyter_client-8.2.0-py3-none-any.whl (103 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m103.2/103.2 kB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting jupyter-core!=5.0.*,>=4.12 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/41/1e/92a67f333b9335f04ce409799c030dcfb291712658b9d9d13997f7c91e5a/jupyter_core-5.3.0-py3-none-any.whl (93 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.2/93.2 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting matplotlib-inline>=0.1 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/a6/2d/2230afd570c70074e80fd06857ba2bdc5f10c055bd9125665fe276fadb67/matplotlib_inline-0.1.3-py3-none-any.whl (8.2 kB)\n",
      "Collecting nest-asyncio (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/06/e0/93453ebab12f5ce9a9ceda2ff71648b30e5f2ce5bba19ee3c95cbd0aaa67/nest_asyncio-1.5.4-py3-none-any.whl (5.1 kB)\n",
      "Collecting packaging (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ec/1a/610693ac4ee14fcdf2d9bf3c493370e4f2ef7ae2e19217d7a237ff42367d/packaging-23.2-py3-none-any.whl (53 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.0/53.0 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting psutil (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/6e/c8/784968329c1c67c28cce91991ef9af8a8913aa5a3399a6a8954b1380572f/psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m280.2/280.2 kB\u001b[0m \u001b[31m28.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pyzmq>=20 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/79/f4/c33ff6e3d7bfbceecbb2176f75328c897365a519f507d226e44eea74d6d2/pyzmq-25.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting tornado>=6.1 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/66/a5/e6da56c03ff61200d5a43cfb75ab09316fc0836aa7ee26b4e9dcbfc3ae85/tornado-6.3.3-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m427.7/427.7 kB\u001b[0m \u001b[31m37.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting traitlets>=5.4.0 (from ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/77/75/c28e9ef7abec2b7e9ff35aea3e0be6c1aceaf7873c26c95ae1f0d594de71/traitlets-5.9.0-py3-none-any.whl (117 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m117.4/117.4 kB\u001b[0m \u001b[31m14.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting backcall (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/4c/1c/ff6546b6c12603d8dd1070aa3c3d273ad4c07f5771689a7b69a550e8c951/backcall-0.2.0-py2.py3-none-any.whl (11 kB)\n",
      "Collecting decorator (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/d5/50/83c593b07763e1161326b3b8c6686f0f4b0f24d5526546bee538c89837d6/decorator-5.1.1-py3-none-any.whl (9.1 kB)\n",
      "Collecting jedi>=0.16 (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/6d/60/4acda63286ef6023515eb914543ba36496b8929cb7af49ecce63afde09c6/jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m24.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting pickleshare (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/9a/41/220f49aaea88bc6fa6cba8d05ecf24676326156c23b991e80b3f2fc24c77/pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)\n",
      "Collecting prompt-toolkit<3.1.0,>=3.0.30 (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/87/3f/1f5a0ff475ae6481f4b0d45d4d911824d3218b94ee2a97a8cb84e5569836/prompt_toolkit-3.0.38-py3-none-any.whl (385 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m385.8/385.8 kB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pygments>=2.4.0 (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/0b/42/d9d95cc461f098f204cd20c85642ae40fbff81f74c300341b8d0e0df14e0/Pygments-2.14.0-py3-none-any.whl (1.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m55.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting stack-data (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/6a/81/aa96c25c27f78cdc444fec27d80f4c05194c591465e491a1358d8a035bc1/stack_data-0.6.2-py3-none-any.whl (24 kB)\n",
      "Collecting pexpect>4.3 (from ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/39/7b/88dbb785881c28a102619d46423cb853b46dbccc70d3ac362d99773a78ce/pexpect-4.8.0-py2.py3-none-any.whl (59 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.0/59.0 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting python-dateutil>=2.8.2 (from jupyter-client>=6.1.12->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m247.7/247.7 kB\u001b[0m \u001b[31m27.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting platformdirs>=2.5 (from jupyter-core!=5.0.*,>=4.12->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ba/24/a83a900a90105f8ad3f20df5bb5a2cde886df7125c7827e196e4ed4fa8a7/platformdirs-3.0.0-py3-none-any.whl (14 kB)\n",
      "Collecting parso<0.9.0,>=0.8.0 (from jedi>=0.16->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/05/63/8011bd08a4111858f79d2b09aad86638490d62fbf881c44e434a6dfca87b/parso-0.8.3-py2.py3-none-any.whl (100 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100.8/100.8 kB\u001b[0m \u001b[31m13.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting ptyprocess>=0.5 (from pexpect>4.3->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)\n",
      "Collecting wcwidth (from prompt-toolkit<3.1.0,>=3.0.30->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/59/7c/e39aca596badaf1b78e8f547c807b04dae603a433d3e7a7e04d67f2ef3e5/wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)\n",
      "Collecting six>=1.5 (from python-dateutil>=2.8.2->jupyter-client>=6.1.12->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (11 kB)\n",
      "Collecting executing>=1.2.0 (from stack-data->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/28/3c/bc3819dd8b1a1588c9215a87271b6178cc5498acaa83885211f5d4d9e693/executing-1.2.0-py2.py3-none-any.whl (24 kB)\n",
      "Collecting asttokens>=2.1.0 (from stack-data->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/f3/e1/64679d9d0759db5b182222c81ff322c2fe2c31e156a59afd6e9208c960e5/asttokens-2.2.1-py2.py3-none-any.whl (26 kB)\n",
      "Collecting pure-eval (from stack-data->ipython>=7.23.1->ipykernel)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/2b/27/77f9d5684e6bce929f5cfe18d6cfbe5133013c06cb2fbf5933670e60761d/pure_eval-0.2.2-py3-none-any.whl (11 kB)\n",
      "Installing collected packages: wcwidth, pure-eval, ptyprocess, pickleshare, executing, backcall, traitlets, tornado, six, pyzmq, pygments, psutil, prompt-toolkit, platformdirs, pexpect, parso, packaging, nest-asyncio, decorator, debugpy, python-dateutil, matplotlib-inline, jupyter-core, jedi, comm, asttokens, stack-data, jupyter-client, ipython, ipykernel\n",
      "Successfully installed asttokens-2.2.1 backcall-0.2.0 comm-0.1.3 debugpy-1.6.6 decorator-5.1.1 executing-1.2.0 ipykernel-6.23.1 ipython-8.10.0 jedi-0.18.2 jupyter-client-8.2.0 jupyter-core-5.3.0 matplotlib-inline-0.1.3 nest-asyncio-1.5.4 packaging-23.2 parso-0.8.3 pexpect-4.8.0 pickleshare-0.7.5 platformdirs-3.0.0 prompt-toolkit-3.0.38 psutil-5.9.4 ptyprocess-0.7.0 pure-eval-0.2.2 pygments-2.14.0 python-dateutil-2.8.2 pyzmq-25.1.1 six-1.16.0 stack-data-0.6.2 tornado-6.3.3 traitlets-5.9.0 wcwidth-0.2.5\n"
     ]
    }
   ],
   "source": [
    "!/home/ma-user/anaconda3/bin/conda create -n python-3.10.10 python=3.10.10 -y --override-channels --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
    "!/home/ma-user/anaconda3/envs/python-3.10.10/bin/pip install ipykernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "59d36081-3b1e-46f1-8c8b-b926349e1092",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "\n",
    "data = {\n",
    "   \"display_name\": \"python-3.10.10\",\n",
    "   \"env\": {\n",
    "      \"PATH\": \"/home/ma-user/anaconda3/envs/python-3.10.10/bin:/home/ma-user/anaconda3/envs/python-3.7.10/bin:/modelarts/authoring/notebook-conda/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/anaconda3/envs/PyTorch-1.8/bin\"\n",
    "   },\n",
    "   \"language\": \"python\",\n",
    "   \"argv\": [\n",
    "      \"/home/ma-user/anaconda3/envs/python-3.10.10/bin/python\",\n",
    "      \"-m\",\n",
    "      \"ipykernel\",\n",
    "      \"-f\",\n",
    "      \"{connection_file}\"\n",
    "   ]\n",
    "}\n",
    "\n",
    "if not os.path.exists(\"/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/\"):\n",
    "    os.mkdir(\"/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/\")\n",
    "\n",
    "with open('/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/kernel.json', 'w') as f:\n",
    "    json.dump(data, f, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c22b7326-10a8-4475-b295-c232961efa9e",
   "metadata": {},
   "source": [
    "### 2.3 安装依赖库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c3c2cd0e-8f3f-4ffa-9a86-2918f0ae15af",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting transformers==4.30.2\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/5b/0b/e45d26ccd28568013523e04f325432ea88a442b4e3020b757cf4361f0120/transformers-4.30.2-py3-none-any.whl (7.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.2/7.2 MB\u001b[0m \u001b[31m98.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting filelock (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/00/45/ec3407adf6f6b5bf867a4462b2b0af27597a26bd3cd6e2534cb6ab029938/filelock-3.12.2-py3-none-any.whl (10 kB)\n",
      "Collecting huggingface-hub<1.0,>=0.14.1 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/a0/0a/02ac0ae1047d97769003ff4fb8e6717024f3f174a5d13257415aa09e13d9/huggingface_hub-0.20.1-py3-none-any.whl (330 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m330.1/330.1 kB\u001b[0m \u001b[31m30.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting numpy>=1.17 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/71/3c/3b1981c6a1986adc9ee7db760c0c34ea5b14ac3da9ecfcf1ea2a4ec6c398/numpy-1.25.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m18.2/18.2 MB\u001b[0m \u001b[31m63.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (23.2)\n",
      "Collecting pyyaml>=5.1 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/29/61/bf33c6c85c55bc45a29eee3195848ff2d518d84735eb0e2d8cb42e0d285e/PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m705.5/705.5 kB\u001b[0m \u001b[31m41.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting regex!=2019.12.17 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/7c/81/b064cc2c67ca2182137641f9d3fd47fe470f1a84674d9b9f91fd39bf0e6f/regex-2023.5.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (769 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m769.7/769.7 kB\u001b[0m \u001b[31m39.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting requests (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl (62 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.6/62.6 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/94/60/ff26cce378023624ffcad91edaa4871f561d6ba7295185c45037ddba80e2/tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m44.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting safetensors>=0.3.1 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/6c/f0/c17bbdb1e5f9dab29d44cade445135789f75f8f08ea2728d04493ea8412b/safetensors-0.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m23.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting tqdm>=4.27 (from transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/00/e5/f12a80907d0884e6dff9c16d0c0114d81b8cd07dc3ae54c5e962cc83037e/tqdm-4.66.1-py3-none-any.whl (78 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.3/78.3 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting fsspec>=2023.5.0 (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/70/25/fab23259a52ece5670dcb8452e1af34b89e6135ecc17cd4b54b4b479eac6/fsspec-2023.12.2-py3-none-any.whl (168 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m169.0/169.0 kB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting typing-extensions>=3.7.4.3 (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/24/21/7d397a4b7934ff4028987914ac1044d3b7d52712f30e2ac7a2ae5bc86dd0/typing_extensions-4.8.0-py3-none-any.whl (31 kB)\n",
      "Collecting charset-normalizer<4,>=2 (from requests->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/da/f1/3702ba2a7470666a62fd81c58a4c40be00670e5006a67f4d626e57f013ae/charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m142.1/142.1 kB\u001b[0m \u001b[31m15.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting idna<4,>=2.5 (from requests->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl (61 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.5/61.5 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting urllib3<3,>=1.21.1 (from requests->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/96/94/c31f58c7a7f470d5665935262ebd7455c7e4c7782eb525658d3dbf4b9403/urllib3-2.1.0-py3-none-any.whl (104 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m104.6/104.6 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting certifi>=2017.4.17 (from requests->transformers==4.30.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/64/62/428ef076be88fa93716b576e4a01f919d25968913e817077a386fcbe4f42/certifi-2023.11.17-py3-none-any.whl (162 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m162.5/162.5 kB\u001b[0m \u001b[31m12.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: tokenizers, safetensors, urllib3, typing-extensions, tqdm, regex, pyyaml, numpy, idna, fsspec, filelock, charset-normalizer, certifi, requests, huggingface-hub, transformers\n",
      "Successfully installed certifi-2023.11.17 charset-normalizer-3.3.2 filelock-3.12.2 fsspec-2023.12.2 huggingface-hub-0.20.1 idna-3.4 numpy-1.25.2 pyyaml-6.0.1 regex-2023.5.5 requests-2.31.0 safetensors-0.3.3 tokenizers-0.13.3 tqdm-4.66.1 transformers-4.30.2 typing-extensions-4.8.0 urllib3-2.1.0\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting sentencepiece==0.1.99\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/7f/e5/323dc813b3e1339305f888d035e2f3725084fc4dcf051995b366dd26cc90/sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m24.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: sentencepiece\n",
      "Successfully installed sentencepiece-0.1.99\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting torch==2.0.1\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/8c/4d/17e07377c9c3d1a0c4eb3fde1c7c16b5a0ce6133ddbabc08ceef6b7f2645/torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m619.9/619.9 MB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: filelock in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (3.12.2)\n",
      "Requirement already satisfied: typing-extensions in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (4.8.0)\n",
      "Collecting sympy (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/d2/05/e6600db80270777c4a64238a98d442f0fd07cc8915be2a1c16da7f2b9e74/sympy-1.12-py3-none-any.whl (5.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.7/5.7 MB\u001b[0m \u001b[31m41.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting networkx (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/42/31/d2f89f1ae42718f8c8a9e440ebe38d7d5fe1e0d9eb9178ce779e365b3ab0/networkx-2.8.8-py3-none-any.whl (2.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m74.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting jinja2 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/bc/c3/f068337a370801f372f2f8f6bad74a5c140f6fda3d9de154052708dd3c65/Jinja2-3.1.2-py3-none-any.whl (133 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m133.1/133.1 kB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cuda-nvrtc-cu11==11.7.99 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ea/8d/0709ba16c2831c17ec1c2ea1eeb89ada11ffa8d966d773cce0a7463b22bb/nvidia_cuda_nvrtc_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (21.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.0/21.0 MB\u001b[0m \u001b[31m81.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cuda-runtime-cu11==11.7.99 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/36/92/89cf558b514125d2ebd8344dd2f0533404b416486ff681d5434a5832a019/nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m849.3/849.3 kB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cuda-cupti-cu11==11.7.101 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/e6/9d/dd0cdcd800e642e3c82ee3b5987c751afd4f3fb9cc2752517f42c3bc6e49/nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.8/11.8 MB\u001b[0m \u001b[31m99.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cudnn-cu11==8.5.0.96 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/dc/30/66d4347d6e864334da5bb1c7571305e501dcb11b9155971421bb7bb5315f/nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m557.1/557.1 MB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cublas-cu11==11.10.3.66 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ce/41/fdeb62b5437996e841d83d7d2714ca75b886547ee8017ee2fe6ea409d983/nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m317.1/317.1 MB\u001b[0m \u001b[31m14.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cufft-cu11==10.9.0.58 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/74/79/b912a77e38e41f15a0581a59f5c3548d1ddfdda3225936fb67c342719e7a/nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m168.4/168.4 MB\u001b[0m \u001b[31m25.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-curand-cu11==10.2.10.91 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/8f/11/af78d54b2420e64a4dd19e704f5bb69dcb5a6a3138b4465d6a48cdf59a21/nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.6/54.6 MB\u001b[0m \u001b[31m55.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cusolver-cu11==11.4.0.1 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/25/4b/272f9aa7838e545b47878e4aec4f09b0fecf17dbd312cf5c5dc398b0637f/nvidia_cusolver_cu11-11.4.0.1-py3-none-manylinux1_x86_64.whl (102.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m102.6/102.6 MB\u001b[0m \u001b[31m38.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cusparse-cu11==11.7.4.91 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ea/6f/6d032cc1bb7db88a989ddce3f4968419a7edeafda362847f42f614b1f845/nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m173.2/173.2 MB\u001b[0m \u001b[31m25.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-nccl-cu11==2.14.3 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/55/92/914cdb650b6a5d1478f83148597a25e90ea37d739bd563c5096b0e8a5f43/nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m177.1/177.1 MB\u001b[0m \u001b[31m26.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-nvtx-cu11==11.7.91 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/23/d5/09493ff0e64fd77523afbbb075108f27a13790479efe86b9ffb4587671b5/nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.6/98.6 kB\u001b[0m \u001b[31m11.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting triton==2.0.0 (from torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ca/31/ff6be541195daf77aa5c72303b2354661a69e717967d44d91eb4f3fdce32/triton-2.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m63.3/63.3 MB\u001b[0m \u001b[31m7.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: setuptools in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (68.2.2)\n",
      "Requirement already satisfied: wheel in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (0.41.2)\n",
      "Collecting cmake (from triton==2.0.0->torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/2e/51/3a4672a819b4532a378bfefad8f886cfe71057556e0d4eefb64523fd370a/cmake-3.27.2-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (26.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m26.1/26.1 MB\u001b[0m \u001b[31m72.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting lit (from triton==2.0.0->torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/63/ab/e4c5f26400970a3ca356d19e592af3a5f05a73d5f35470accb6600f84045/lit-16.0.5.tar.gz (138 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.0/138.0 kB\u001b[0m \u001b[31m12.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting MarkupSafe>=2.0 (from jinja2->torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/12/b3/d9ed2c0971e1435b8a62354b18d3060b66c8cb1d368399ec0b9baa7c0ee5/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)\n",
      "Collecting mpmath>=0.19 (from sympy->torch==2.0.1)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl (536 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.2/536.2 kB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hBuilding wheels for collected packages: lit\n",
      "  Building wheel for lit (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for lit: filename=lit-16.0.5-py3-none-any.whl size=88176 sha256=6217a85f83d85aa2063c647b7392dd8dcf1264f1b502e9ca6dd48d6cd8f66e8d\n",
      "  Stored in directory: /home/ma-user/.cache/pip/wheels/e1/dc/f7/0eb9b51ba2fcb3a439d56ce1259dc815e5bcdd329c2b874f34\n",
      "Successfully built lit\n",
      "Installing collected packages: mpmath, lit, cmake, sympy, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-cusolver-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, networkx, MarkupSafe, nvidia-cudnn-cu11, jinja2, triton, torch\n",
      "Successfully installed MarkupSafe-2.1.3 cmake-3.27.2 jinja2-3.1.2 lit-16.0.5 mpmath-1.3.0 networkx-2.8.8 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 sympy-1.12 torch-2.0.1 triton-2.0.0\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting markdown==3.4.3\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/9a/a1/1352b0e5a3c71a79fa9265726e2217f69df9fd4de0bcb5725cc61f62a5df/Markdown-3.4.3-py3-none-any.whl (93 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.9/93.9 kB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: markdown\n",
      "Successfully installed markdown-3.4.3\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting faiss-gpu==1.7.2\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/a8/71/623896382d90a9a99adf3438aa2c575535ba37804be9701d66f3337afd83/faiss_gpu-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (85.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m85.5/85.5 MB\u001b[0m \u001b[31m42.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: faiss-gpu\n",
      "Successfully installed faiss-gpu-1.7.2\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting langchain==0.0.329\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/42/4e/86204994aeb2e4ac367a7fade896b13532eae2430299052eb2c80ca35d2c/langchain-0.0.329-py3-none-any.whl (2.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m47.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: PyYAML>=5.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langchain==0.0.329) (6.0.1)\n",
      "Collecting SQLAlchemy<3,>=1.4 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/19/fc/a382d17e2ac9d94a9d8fd534de666638c488e384433b57a3350e046a0270/SQLAlchemy-2.0.24-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.1/3.1 MB\u001b[0m \u001b[31m83.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hCollecting aiohttp<4.0.0,>=3.8.3 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2f/16/50441c4baa39e5426181c6f630203ab65029f9a9c55d0a1019a31c26d702/aiohttp-3.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m62.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting anyio<4.0 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl (80 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m80.9/80.9 kB\u001b[0m \u001b[31m10.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting async-timeout<5.0.0,>=4.0.0 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a7/fa/e01228c2938de91d47b307831c62ab9e4001e747789d0b05baf779a6488c/async_timeout-4.0.3-py3-none-any.whl (5.7 kB)\n",
      "Collecting dataclasses-json<0.7,>=0.5.7 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/53/8c006de775834cd4ea64a445402dc195caeebb77dc76b7defb9b3887cb0d/dataclasses_json-0.6.3-py3-none-any.whl (28 kB)\n",
      "Collecting jsonpatch<2.0,>=1.33 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/73/07/02e16ed01e04a374e644b575638ec7987ae846d25ad97bcc9945a3ee4b0e/jsonpatch-1.33-py2.py3-none-any.whl (12 kB)\n",
      "Collecting langsmith<0.1.0,>=0.0.52 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c0/a2/7814b2341d2919f8305cdaff2e37f76b04c45839f402a38cf13ef7153bea/langsmith-0.0.75-py3-none-any.whl (46 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.7/46.7 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: numpy<2,>=1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langchain==0.0.329) (1.25.2)\n",
      "Collecting pydantic<3,>=1 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/dd/b7/9aea7ee6c01fe3f3c03b8ca3c7797c866df5fecece9d6cb27caa138db2e2/pydantic-2.5.3-py3-none-any.whl (381 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m381.9/381.9 kB\u001b[0m \u001b[31m34.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: requests<3,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langchain==0.0.329) (2.31.0)\n",
      "Collecting tenacity<9.0.0,>=8.1.0 (from langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f4/f1/990741d5bb2487d529d20a433210ffa136a367751e454214013b441c4575/tenacity-8.2.3-py3-none-any.whl (24 kB)\n",
      "Collecting attrs>=17.3.0 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f0/eb/fcb708c7bf5056045e9e98f62b93bd7467eb718b0202e7698eb11d66416c/attrs-23.1.0-py3-none-any.whl (61 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.2/61.2 kB\u001b[0m \u001b[31m8.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting multidict<7.0,>=4.5 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/56/b5/ac112889bfc68e6cf4eda1e4325789b166c51c6cd29d5633e28fb2c2f966/multidict-6.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.5/114.5 kB\u001b[0m \u001b[31m15.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting yarl<2.0,>=1.0 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c3/a0/0ade1409d184cbc9e85acd403a386a7c0563b92ff0f26d138ff9e86e48b4/yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m301.6/301.6 kB\u001b[0m \u001b[31m31.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting frozenlist>=1.1.1 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ec/25/0c87df2e53c0c5d90f7517ca0ff7aca78d050a8ec4d32c4278e8c0e52e51/frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m239.5/239.5 kB\u001b[0m \u001b[31m15.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting aiosignal>=1.1.2 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/76/ac/a7305707cb852b7e16ff80eaf5692309bde30e2b1100a1fcacdc8f731d97/aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
      "Requirement already satisfied: idna>=2.8 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from anyio<4.0->langchain==0.0.329) (3.4)\n",
      "Collecting sniffio>=1.1 (from anyio<4.0->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c3/a0/5dba8ed157b0136607c7f2151db695885606968d1fae123dc3391e0cfdbf/sniffio-1.3.0-py3-none-any.whl (10 kB)\n",
      "Collecting exceptiongroup (from anyio<4.0->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b8/9a/5028fd52db10e600f1c4674441b968cf2ea4959085bfb5b99fb1250e5f68/exceptiongroup-1.2.0-py3-none-any.whl (16 kB)\n",
      "Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ed/3c/cebfdcad015240014ff08b883d1c0c427f2ba45ae8c6572851b6ef136cad/marshmallow-3.20.1-py3-none-any.whl (49 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting typing-inspect<1,>=0.4.0 (from dataclasses-json<0.7,>=0.5.7->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/65/f3/107a22063bf27bdccf2024833d3445f4eea42b2e598abfbd46f6a63b6cb0/typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n",
      "Collecting jsonpointer>=1.9 (from jsonpatch<2.0,>=1.33->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/12/f6/0232cc0c617e195f06f810534d00b74d2f348fe71b2118009ad8ad31f878/jsonpointer-2.4-py2.py3-none-any.whl (7.8 kB)\n",
      "Collecting annotated-types>=0.4.0 (from pydantic<3,>=1->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/28/78/d31230046e58c207284c6b2c4e8d96e6d3cb4e52354721b944d3e1ee4aa5/annotated_types-0.6.0-py3-none-any.whl (12 kB)\n",
      "Collecting pydantic-core==2.14.6 (from pydantic<3,>=1->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/90/28/3c6843e6b203999be2660d3f114be196f2182dcac533dc764ad320c9184d/pydantic_core-2.14.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: typing-extensions>=4.6.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from pydantic<3,>=1->langchain==0.0.329) (4.8.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.329) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.329) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.329) (2023.11.17)\n",
      "Collecting greenlet!=0.4.17 (from SQLAlchemy<3,>=1.4->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/24/35/945d5b10648fec9b20bcc6df8952d20bb3bba76413cd71c1fdbee98f5616/greenlet-3.0.3-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (616 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m616.0/616.0 kB\u001b[0m \u001b[31m44.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: packaging>=17.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json<0.7,>=0.5.7->langchain==0.0.329) (23.2)\n",
      "Collecting mypy-extensions>=0.3.0 (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain==0.0.329)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2a/e2/5d3f6ada4297caebe1a2add3b126fe800c96f56dbe5d1988a2cbe0b267aa/mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
      "Installing collected packages: tenacity, sniffio, pydantic-core, mypy-extensions, multidict, marshmallow, jsonpointer, greenlet, frozenlist, exceptiongroup, attrs, async-timeout, annotated-types, yarl, typing-inspect, SQLAlchemy, pydantic, jsonpatch, anyio, aiosignal, langsmith, dataclasses-json, aiohttp, langchain\n",
      "Successfully installed SQLAlchemy-2.0.24 aiohttp-3.9.1 aiosignal-1.3.1 annotated-types-0.6.0 anyio-3.7.1 async-timeout-4.0.3 attrs-23.1.0 dataclasses-json-0.6.3 exceptiongroup-1.2.0 frozenlist-1.4.1 greenlet-3.0.3 jsonpatch-1.33 jsonpointer-2.4 langchain-0.0.329 langsmith-0.0.75 marshmallow-3.20.1 multidict-6.0.4 mypy-extensions-1.0.0 pydantic-2.5.3 pydantic-core-2.14.6 sniffio-1.3.0 tenacity-8.2.3 typing-inspect-0.9.0 yarl-1.9.4\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting nltk==3.8.1\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a6/0a/0d20d2c0f16be91b9fa32a77b76c60f9baf6eba419e5ef5deca17af9c582/nltk-3.8.1-py3-none-any.whl (1.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m25.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting click (from nltk==3.8.1)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl (97 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.9/97.9 kB\u001b[0m \u001b[31m8.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting joblib (from nltk==3.8.1)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/10/40/d551139c85db202f1f384ba8bcf96aca2f329440a844f924c8a0040b6d02/joblib-1.3.2-py3-none-any.whl (302 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.2/302.2 kB\u001b[0m \u001b[31m25.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: regex>=2021.8.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk==3.8.1) (2023.5.5)\n",
      "Requirement already satisfied: tqdm in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk==3.8.1) (4.66.1)\n",
      "Installing collected packages: joblib, click, nltk\n",
      "Successfully installed click-8.1.7 joblib-1.3.2 nltk-3.8.1\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting unstructured==0.10.24\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/5d/10/f36ec76d07acee92ea7432c00e41d15ebf47df1db994778a1710da77e4a2/unstructured-0.10.24-py3-none-any.whl (1.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m38.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting chardet (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/38/6f/f5fbc992a329ee4e0f288c1fe0e2ad9485ed064cac731ed2fe47dcc38cbf/chardet-5.2.0-py3-none-any.whl (199 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m199.4/199.4 kB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting filetype (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/18/79/1b8fa1bb3568781e84c9200f951c735f3f157429f44be0495da55894d620/filetype-1.2.0-py2.py3-none-any.whl (19 kB)\n",
      "Collecting python-magic (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6c/73/9f872cb81fc5c3bb48f7227872c28975f998f3e7c2b1c16e95e6432bbb90/python_magic-0.4.27-py2.py3-none-any.whl (13 kB)\n",
      "Collecting lxml (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/57/a0/10b7b263f37a109c88d38bf8c20bae5d03b13c06cd7c7aa1aff985b04e4b/lxml-5.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (7.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m91.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: nltk in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from unstructured==0.10.24) (3.8.1)\n",
      "Collecting tabulate (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/40/44/4a5f08c96eb108af5cb50b41f76142f0afa346dfa99d5296fe7202a11854/tabulate-0.9.0-py3-none-any.whl (35 kB)\n",
      "Requirement already satisfied: requests in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from unstructured==0.10.24) (2.31.0)\n",
      "Collecting beautifulsoup4 (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/57/f4/a69c20ee4f660081a7dedb1ac57f29be9378e04edfcb90c526b923d4bebc/beautifulsoup4-4.12.2-py3-none-any.whl (142 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m143.0/143.0 kB\u001b[0m \u001b[31m16.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting emoji (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/03/40/91d0c9fe5a0b494c0fdbcacda4d203aea39f8293e69c70129389308ca928/emoji-2.9.0-py2.py3-none-any.whl (397 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m397.5/397.5 kB\u001b[0m \u001b[31m36.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: dataclasses-json in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from unstructured==0.10.24) (0.6.3)\n",
      "Collecting python-iso639 (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d4/3e/f20c6edf8b95db750a52269d3dffd6e6a4432989cc7731efef2a06c1d187/python_iso639-2023.12.11-py3-none-any.whl (275 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m275.1/275.1 kB\u001b[0m \u001b[31m28.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting langdetect (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/0e/72/a3add0e4eec4eb9e2569554f7c70f4a3c27712f40e3284d483e88094cc0e/langdetect-1.0.9.tar.gz (981 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m981.5/981.5 kB\u001b[0m \u001b[31m58.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: numpy in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from unstructured==0.10.24) (1.25.2)\n",
      "Collecting rapidfuzz (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/02/39/3f94121e21b78e0a2699b272a8906ee5eb6f9d70082d90784464b0a4fcc8/rapidfuzz-3.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m85.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hCollecting backoff (from unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/df/73/b6e24bd22e6720ca8ee9a85a0c4a2971af8497d8f3193fa05390cbd46e09/backoff-2.2.1-py3-none-any.whl (15 kB)\n",
      "Collecting soupsieve>1.2 (from beautifulsoup4->unstructured==0.10.24)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/4c/f3/038b302fdfbe3be7da016777069f26ceefe11a681055ea1f7817546508e3/soupsieve-2.5-py3-none-any.whl (36 kB)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from dataclasses-json->unstructured==0.10.24) (3.20.1)\n",
      "Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from dataclasses-json->unstructured==0.10.24) (0.9.0)\n",
      "Requirement already satisfied: six in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langdetect->unstructured==0.10.24) (1.16.0)\n",
      "Requirement already satisfied: click in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->unstructured==0.10.24) (8.1.7)\n",
      "Requirement already satisfied: joblib in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->unstructured==0.10.24) (1.3.2)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->unstructured==0.10.24) (2023.5.5)\n",
      "Requirement already satisfied: tqdm in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->unstructured==0.10.24) (4.66.1)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->unstructured==0.10.24) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->unstructured==0.10.24) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->unstructured==0.10.24) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->unstructured==0.10.24) (2023.11.17)\n",
      "Requirement already satisfied: packaging>=17.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->unstructured==0.10.24) (23.2)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json->unstructured==0.10.24) (1.0.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json->unstructured==0.10.24) (4.8.0)\n",
      "Building wheels for collected packages: langdetect\n",
      "  Building wheel for langdetect (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for langdetect: filename=langdetect-1.0.9-py3-none-any.whl size=993224 sha256=4f117677fa47ece224ae89ac28530922433c0d98b243463f3e1262ecb1c0b460\n",
      "  Stored in directory: /home/ma-user/.cache/pip/wheels/ee/e9/63/fe12d571f8675325c5e131236f64a52b7ed05da124bd628a74\n",
      "Successfully built langdetect\n",
      "Installing collected packages: filetype, tabulate, soupsieve, rapidfuzz, python-magic, python-iso639, lxml, langdetect, emoji, chardet, backoff, beautifulsoup4, unstructured\n",
      "Successfully installed backoff-2.2.1 beautifulsoup4-4.12.2 chardet-5.2.0 emoji-2.9.0 filetype-1.2.0 langdetect-1.0.9 lxml-5.0.0 python-iso639-2023.12.11 python-magic-0.4.27 rapidfuzz-3.6.1 soupsieve-2.5 tabulate-0.9.0 unstructured-0.10.24\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting sentence-transformers==2.2.2\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/20/9c/f07bd70d128fdb107bc02a0c702b9058b4fe147d0ba67b5a0f4c3cf15a54/sentence-transformers-2.2.2.tar.gz (85 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.0/86.0 kB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: transformers<5.0.0,>=4.6.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (4.30.2)\n",
      "Requirement already satisfied: tqdm in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (4.66.1)\n",
      "Requirement already satisfied: torch>=1.6.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (2.0.1)\n",
      "Collecting torchvision (from sentence-transformers==2.2.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/87/0f/88f023bf6176d9af0f85feedf4be129f9cf2748801c4d9c690739a10c100/torchvision-0.15.2-cp310-cp310-manylinux1_x86_64.whl (6.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.0/6.0 MB\u001b[0m \u001b[31m89.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: numpy in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (1.25.2)\n",
      "Collecting scikit-learn (from sentence-transformers==2.2.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/5c/e9/ee572691a3fb05555bcde41826faad29ae4bc1fb07982e7f53d54a176879/scikit_learn-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.8/10.8 MB\u001b[0m \u001b[31m57.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m0:01\u001b[0m\n",
      "\u001b[?25hCollecting scipy (from sentence-transformers==2.2.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/a8/cc/c36f3439f5d47c3b13833ce6687b43a040cc7638c502ac46b41e2d4f3d6f/scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m36.3/36.3 MB\u001b[0m \u001b[31m48.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: nltk in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (3.8.1)\n",
      "Requirement already satisfied: sentencepiece in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (0.1.99)\n",
      "Requirement already satisfied: huggingface-hub>=0.4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sentence-transformers==2.2.2) (0.20.1)\n",
      "Requirement already satisfied: filelock in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (3.12.2)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2023.12.2)\n",
      "Requirement already satisfied: requests in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2.31.0)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (6.0.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (4.8.0)\n",
      "Requirement already satisfied: packaging>=20.9 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (23.2)\n",
      "Requirement already satisfied: sympy in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (1.12)\n",
      "Requirement already satisfied: networkx in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (2.8.8)\n",
      "Requirement already satisfied: jinja2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (3.1.2)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.7.101)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (8.5.0.96)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.10.3.66)\n",
      "Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (10.9.0.58)\n",
      "Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (10.2.10.91)\n",
      "Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.4.0.1)\n",
      "Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.7.4.91)\n",
      "Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (2.14.3)\n",
      "Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (11.7.91)\n",
      "Requirement already satisfied: triton==2.0.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2) (2.0.0)\n",
      "Requirement already satisfied: setuptools in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.6.0->sentence-transformers==2.2.2) (68.2.2)\n",
      "Requirement already satisfied: wheel in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.6.0->sentence-transformers==2.2.2) (0.41.2)\n",
      "Requirement already satisfied: cmake in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from triton==2.0.0->torch>=1.6.0->sentence-transformers==2.2.2) (3.27.2)\n",
      "Requirement already satisfied: lit in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from triton==2.0.0->torch>=1.6.0->sentence-transformers==2.2.2) (16.0.5)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2) (2023.5.5)\n",
      "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2) (0.13.3)\n",
      "Requirement already satisfied: safetensors>=0.3.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2) (0.3.3)\n",
      "Requirement already satisfied: click in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->sentence-transformers==2.2.2) (8.1.7)\n",
      "Requirement already satisfied: joblib in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nltk->sentence-transformers==2.2.2) (1.3.2)\n",
      "Collecting threadpoolctl>=2.0.0 (from scikit-learn->sentence-transformers==2.2.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/61/cf/6e354304bcb9c6413c4e02a747b600061c21d38ba51e7e544ac7bc66aecc/threadpoolctl-3.1.0-py3-none-any.whl (14 kB)\n",
      "Collecting pillow!=8.3.*,>=5.3.0 (from torchvision->sentence-transformers==2.2.2)\n",
      "  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/95/7b/71e2665760b5c33af00fa9bb6d6bca068b51bf021a4ceaeee03e18689f51/Pillow-10.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.5/3.5 MB\u001b[0m \u001b[31m33.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: MarkupSafe>=2.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from jinja2->torch>=1.6.0->sentence-transformers==2.2.2) (2.1.3)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers==2.2.2) (2023.11.17)\n",
      "Requirement already satisfied: mpmath>=0.19 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sympy->torch>=1.6.0->sentence-transformers==2.2.2) (1.3.0)\n",
      "Building wheels for collected packages: sentence-transformers\n",
      "  Building wheel for sentence-transformers (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125923 sha256=5b770c4764866d2a4c1bd8f46365e30710ad9b25e123f37f422611bdecb0bd59\n",
      "  Stored in directory: /home/ma-user/.cache/pip/wheels/4f/9d/a5/9beabf87fdb3e143ae061b7bd2356c98c21bae0c908df108ee\n",
      "Successfully built sentence-transformers\n",
      "Installing collected packages: threadpoolctl, scipy, pillow, scikit-learn, torchvision, sentence-transformers\n",
      "Successfully installed pillow-10.1.0 scikit-learn-1.3.0 scipy-1.11.2 sentence-transformers-2.2.2 threadpoolctl-3.1.0 torchvision-0.15.2\n",
      "cp: cannot stat '/home/ma-user/work/frpc_linux_amd64': No such file or directory\n",
      "chmod: cannot access '/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/gradio/frpc_linux_amd64_v0.2': No such file or directory\n"
     ]
    }
   ],
   "source": [
    "!pip install transformers==4.30.2\n",
    "!pip install sentencepiece==0.1.99\n",
    "!pip install torch==2.0.1\n",
    "!pip install markdown==3.4.3\n",
    "!pip install faiss-gpu==1.7.2\n",
    "!pip install langchain==0.0.329 -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "!pip install nltk==3.8.1 -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "!pip install unstructured==0.10.24 -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "!pip install sentence-transformers==2.2.2\n",
    "!cp /home/ma-user/work/frpc_linux_amd64 /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/gradio/frpc_linux_amd64_v0.2\n",
    "!chmod +x /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/gradio/frpc_linux_amd64_v0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f1d88281-0864-4b5c-8a5f-ec655bb41902",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting rapidocr_onnxruntime\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/06/bd/058ea31f987eab3160c40b88ee28918813d42054d1261a7fd8019aed05db/rapidocr_onnxruntime-1.3.9-py3-none-any.whl (14.9 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.9/14.9 MB\u001b[0m \u001b[31m83.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting pdf2image\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e2/8c/37a7467d88c47e435429c147020c2fb53206fabfc44b8b72f9728e79ce1a/pdf2image-1.16.3-py3-none-any.whl (11 kB)\n",
      "Collecting pdfminer.six\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/eb/9c/e46fe7502b32d7db6af6e36a9105abb93301fa1ec475b5ddcba8b35ae23a/pdfminer.six-20231228-py3-none-any.whl (5.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.6/5.6 MB\u001b[0m \u001b[31m88.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting mdtex2html\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/47/fa/5156a032ad68f6c32ae0dc3aaf8b3d690004b42497d4735a08bb4cea6ec3/mdtex2html-1.2.0-py3-none-any.whl (13 kB)\n",
      "Collecting gradio==3.39\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/82/5f/c815ae438b63ca8b7418acf470369493bac5aa267192702c8601ca67966b/gradio-3.39.0-py3-none-any.whl (19.9 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m19.9/19.9 MB\u001b[0m \u001b[31m83.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting aiofiles<24.0,>=22.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c5/19/5af6804c4cc0fed83f47bff6e413a98a36618e7d40185cd36e69737f3b0e/aiofiles-23.2.1-py3-none-any.whl (15 kB)\n",
      "Requirement already satisfied: aiohttp~=3.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (3.9.1)\n",
      "Collecting altair<6.0,>=4.2.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c5/e4/7fcceef127badbb0d644d730d992410e4f3799b295c9964a172f92a469c7/altair-5.2.0-py3-none-any.whl (996 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m996.9/996.9 kB\u001b[0m \u001b[31m55.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting fastapi (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d4/e0/d5d6482e992a1892f3a9a62f6a9154944ae5b276e7da1cf92faa02e3a107/fastapi-0.108.0-py3-none-any.whl (92 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.0/92.0 kB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting ffmpy (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9a/06/49b275a312eb207e2a2718a7414dedfded05088437352b67aaa9a355f948/ffmpy-0.3.1.tar.gz (5.5 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting gradio-client>=0.3.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c4/e7/5da3a4b6108f5e2e43d034d7923c3562f93beba8f3d13a0ec7c201a6f33c/gradio_client-0.8.0-py3-none-any.whl (305 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m305.1/305.1 kB\u001b[0m \u001b[31m30.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting httpx (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/39/9b/4937d841aee9c2c8102d9a4eeb800c7dad25386caabb4a1bf5010df81a57/httpx-0.26.0-py3-none-any.whl (75 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.9/75.9 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: huggingface-hub>=0.14.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (0.20.1)\n",
      "Requirement already satisfied: jinja2<4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (3.1.2)\n",
      "Collecting markdown-it-py>=2.0.0 (from markdown-it-py[linkify]>=2.0.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/42/d7/1ec15b46af6af88f19b8e5ffea08fa375d433c998b8a7639e76935c14f1f/markdown_it_py-3.0.0-py3-none-any.whl (87 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m87.5/87.5 kB\u001b[0m \u001b[31m10.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: markupsafe~=2.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (2.1.3)\n",
      "Collecting matplotlib~=3.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/19/e5/a4ea514515f270224435c69359abb7a3d152ed31b9ee3ba5e63017461945/matplotlib-3.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.6/11.6 MB\u001b[0m \u001b[31m99.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting mdit-py-plugins<=0.3.3 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/33/eb/c358112e8265f827cf8228eda36cf2a720ad933f5ca66f47f808edf4bb34/mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.5/50.5 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: numpy~=1.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (1.25.2)\n",
      "Collecting orjson~=3.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/17/e2/7ff96963ba854f0a807fd2783bd7d947ecb0cac7df1d802699727c418aec/orjson-3.9.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.7/138.7 kB\u001b[0m \u001b[31m16.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: packaging in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (23.2)\n",
      "Collecting pandas<3.0,>=1.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b1/67/aca1f6e215d957d24d0a290321f368503305480268f9617bf625243e9dea/pandas-2.1.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.3/12.3 MB\u001b[0m \u001b[31m96.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: pillow<11.0,>=8.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (10.1.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (2.5.3)\n",
      "Collecting pydub (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a6/53/d78dc063216e62fc55f6b2eebb447f6a4b0a59f55c8406376f76bf959b08/pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
      "Collecting python-multipart (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b4/ff/b1e11d8bffb5e0e1b6d27f402eeedbeb9be6df2cdbc09356a1ae49806dbf/python_multipart-0.0.6-py3-none-any.whl (45 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.7/45.7 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: pyyaml<7.0,>=5.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (6.0.1)\n",
      "Requirement already satisfied: requests~=2.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (2.31.0)\n",
      "Collecting semantic-version~=2.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6a/23/8146aad7d88f4fcb3a6218f41a60f6c2d4e3a72de72da1825dc7c8f7877c/semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
      "Requirement already satisfied: typing-extensions~=4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (4.8.0)\n",
      "Collecting uvicorn>=0.14.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/26/59/fddd9df489fe27f492cc97626e03663fb3b9b6ef7ce8597a7cdc5f2cbbad/uvicorn-0.25.0-py3-none-any.whl (60 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.3/60.3 kB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting websockets<12.0,>=10.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/58/0a/7570e15661a0a546c3a1152d95fe8c05480459bab36247f0acbf41f01a41/websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m16.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pyclipper>=1.2.0 (from rapidocr_onnxruntime)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/1c/81/4aa8403e587a4c60e00b479c11254a6e3200f3b985dcf4caecf0d8c21261/pyclipper-1.3.0.post5-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (908 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m908.3/908.3 kB\u001b[0m \u001b[31m57.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting onnxruntime>=1.7.0 (from rapidocr_onnxruntime)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7a/cf/6aa8c56fd63f53c2c485921e411269c7b501a2b4e634bd02f226ab2d5d8e/onnxruntime-1.16.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.4/6.4 MB\u001b[0m \u001b[31m96.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting opencv-python>=4.5.1.48 (from rapidocr_onnxruntime)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b7/8a/b2f7e1a434d56bf1d7570fc5941ace0847404e1032d7f1f0b8fed896568d/opencv_python-4.8.1.78-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.7/61.7 MB\u001b[0m \u001b[31m50.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: six>=1.15.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from rapidocr_onnxruntime) (1.16.0)\n",
      "Collecting Shapely>=1.7.1 (from rapidocr_onnxruntime)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/99/e9/a996a080d8478f4ab5ea82f64a5f39aaa8e05c99c2703e0ee03ec8c9e924/shapely-2.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.5/2.5 MB\u001b[0m \u001b[31m86.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pillow<11.0,>=8.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/52/97/9fb1b66fbf98893722247fdf6ac39467a7b177483b546a94dbcff66653f0/Pillow-10.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.5/3.5 MB\u001b[0m \u001b[31m88.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: charset-normalizer>=2.0.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from pdfminer.six) (3.3.2)\n",
      "Collecting cryptography>=36.0.0 (from pdfminer.six)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/14/fd/dd5bd6ab0d12476ebca579cbfd48d31bd90fa28fa257b209df585dcf62a0/cryptography-41.0.7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.4/4.4 MB\u001b[0m \u001b[31m96.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: markdown in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from mdtex2html) (3.4.3)\n",
      "Collecting latex2mathml (from mdtex2html)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f2/0a/181ed55562ce90179aedf33b09fcd79db31c868a5d480f3cb71a31d19692/latex2mathml-3.77.0-py3-none-any.whl (73 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m73.7/73.7 kB\u001b[0m \u001b[31m8.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: attrs>=17.3.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (23.1.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (1.9.4)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (1.4.1)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (1.3.1)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (4.0.3)\n",
      "Collecting jsonschema>=3.0 (from altair<6.0,>=4.2.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/0f/ed/0058234d8dd2b1fc6beeea8eab945191a05e9d391a63202f49fe23327586/jsonschema-4.20.0-py3-none-any.whl (84 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.7/84.7 kB\u001b[0m \u001b[31m10.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting toolz (from altair<6.0,>=4.2.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7f/5c/922a3508f5bda2892be3df86c74f9cf1e01217c2b1f8a0ac4841d903e3e9/toolz-0.12.0-py3-none-any.whl (55 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.8/55.8 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting cffi>=1.12 (from cryptography>=36.0.0->pdfminer.six)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c9/7c/43d81bdd5a915923c3bad5bb4bff401ea00ccc8e28433fb6083d2e3bf58e/cffi-1.16.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (443 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m443.9/443.9 kB\u001b[0m \u001b[31m39.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: fsspec in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio-client>=0.3.0->gradio==3.39) (2023.12.2)\n",
      "Requirement already satisfied: filelock in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.14.0->gradio==3.39) (3.12.2)\n",
      "Requirement already satisfied: tqdm>=4.42.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.14.0->gradio==3.39) (4.66.1)\n",
      "Collecting mdurl~=0.1 (from markdown-it-py>=2.0.0->markdown-it-py[linkify]>=2.0.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl (10.0 kB)\n",
      "Collecting linkify-it-py<3,>=1 (from markdown-it-py[linkify]>=2.0.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/1f/1a/16b0d2f66601ba3081f1d4177087c79fd1f11d17706ee01d373e4ba8e00d/linkify_it_py-2.0.2-py3-none-any.whl (19 kB)\n",
      "Collecting contourpy>=1.0.1 (from matplotlib~=3.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/58/56/e2c43dcfa1f9c7db4d5e3d6f5134b24ed953f4e2133a4b12f0062148db58/contourpy-1.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (310 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m310.7/310.7 kB\u001b[0m \u001b[31m26.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting cycler>=0.10 (from matplotlib~=3.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl (8.3 kB)\n",
      "Collecting fonttools>=4.22.0 (from matplotlib~=3.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/76/2b/2e2c542027ec75956a882e50ca13edbd76f1079fe640b4ae94a782c78ae6/fonttools-4.47.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.6/4.6 MB\u001b[0m \u001b[31m92.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hCollecting kiwisolver>=1.3.1 (from matplotlib~=3.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6f/40/4ab1fdb57fced80ce5903f04ae1aed7c1d5939dda4fd0c0aa526c12fe28a/kiwisolver-1.4.5-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m75.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pyparsing>=2.3.1 (from matplotlib~=3.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/39/92/8486ede85fcc088f1b3dba4ce92dd29d126fd96b0008ea213167940a2475/pyparsing-3.1.1-py3-none-any.whl (103 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m103.1/103.1 kB\u001b[0m \u001b[31m10.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: python-dateutil>=2.7 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.39) (2.8.2)\n",
      "INFO: pip is looking at multiple versions of mdit-py-plugins to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting mdit-py-plugins<=0.3.3 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e4/92/268849737427f7f9a128d2586e561a7dfac723c02f3aaaaef4d27dd6829c/mdit_py_plugins-0.3.2-py3-none-any.whl (50 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.4/50.4 kB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/de/d9/20870f611989b8dcfd2395eddefdd4b1983d6c36513cce7fbbe9eb345768/mdit_py_plugins-0.3.1-py3-none-any.whl (46 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.5/46.5 kB\u001b[0m \u001b[31m5.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/5b/c4/1cf60e11b55197fa2e5e8d2f732a229690f5a08b018ae1cf4c00585ca834/mdit_py_plugins-0.3.0-py3-none-any.whl (43 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.7/43.7 kB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c0/cb/782222da2cc3d543aee662c33cbaf611ec010146ca21c91d5743e8d99603/mdit_py_plugins-0.2.8-py3-none-any.whl (41 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.0/41.0 kB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/58/04/d5538523c5aae983c6057e6feba23dccb1da39d022264d7dacac1bfc4cd6/mdit_py_plugins-0.2.7-py3-none-any.whl (41 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.0/41.0 kB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/0c/31/f0ecaccf7cd2db17332a94852f190840167c3cb7eadf09efe498412f909a/mdit_py_plugins-0.2.6-py3-none-any.whl (39 kB)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/66/62/fae9ee5766a7153d571ad732ef514c552efeaa31735fd60e6d9bc07fa9e4/mdit_py_plugins-0.2.5-py3-none-any.whl (39 kB)\n",
      "INFO: pip is still looking at multiple versions of mdit-py-plugins to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9d/df/16e87ebd0bb9d946d3b2d39c1171398f7f71eef5d9ca85adb94131b8c7b5/mdit_py_plugins-0.2.4-py3-none-any.whl (39 kB)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/97/23/3e017136aefe61b442fee2869fcc7756c814e3cfe551f225e1e085f27217/mdit_py_plugins-0.2.3-py3-none-any.whl (39 kB)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/40/fe/0f06e8545ac0578741342cc00305b8d955863b72755af0c3771f339099fb/mdit_py_plugins-0.2.2-py3-none-any.whl (39 kB)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/64/f2/b477dfb13e6d3db6421cf61cd7b76562f16141e81cb174d837dda519253b/mdit_py_plugins-0.2.1-py3-none-any.whl (38 kB)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/5b/78/e3b8c43edce725d01021d5b2947dc6d425ee5201cece4bc3d6a757720151/mdit_py_plugins-0.2.0-py3-none-any.whl (38 kB)\n",
      "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/00/02/280276967a2328199700b5e16bf427c4b4af1c54fdd6a00623ef93ed25c2/mdit_py_plugins-0.1.0-py3-none-any.whl (37 kB)\n",
      "Collecting markdown-it-py[linkify]>=2.0.0 (from gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/bf/25/2d88e8feee8e055d015343f9b86e370a1ccbec546f2865c98397aaef24af/markdown_it_py-2.2.0-py3-none-any.whl (84 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.5/84.5 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting coloredlogs (from onnxruntime>=1.7.0->rapidocr_onnxruntime)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a7/06/3d6badcf13db419e25b07041d9c7b4a2c331d3f4e7134445ec5df57714cd/coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting flatbuffers (from onnxruntime>=1.7.0->rapidocr_onnxruntime)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6f/12/d5c79ee252793ffe845d58a913197bfa02ae9a0b5c9bc3dc4b58d477b9e7/flatbuffers-23.5.26-py2.py3-none-any.whl (26 kB)\n",
      "Collecting protobuf (from onnxruntime>=1.7.0->rapidocr_onnxruntime)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/5b/7ed02a9b8e752c8f7bca8661779c0275b9e3e6a903a3045e6da51f796dda/protobuf-4.25.1-cp37-abi3-manylinux2014_x86_64.whl (294 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m294.6/294.6 kB\u001b[0m \u001b[31m28.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: sympy in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from onnxruntime>=1.7.0->rapidocr_onnxruntime) (1.12)\n",
      "Collecting pytz>=2020.1 (from pandas<3.0,>=1.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/32/4d/aaf7eff5deb402fd9a24a1449a8119f00d74ae9c2efa79f8ef9994261fc2/pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m502.5/502.5 kB\u001b[0m \u001b[31m43.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tzdata>=2022.1 (from pandas<3.0,>=1.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a3/fb/52b62131e21b24ee297e4e95ed41eba29647dad0e0051a92bb66b43c70ff/tzdata-2023.4-py2.py3-none-any.whl (346 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m346.6/346.6 kB\u001b[0m \u001b[31m27.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: annotated-types>=0.4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4->gradio==3.39) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.14.6 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4->gradio==3.39) (2.14.6)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests~=2.0->gradio==3.39) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests~=2.0->gradio==3.39) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests~=2.0->gradio==3.39) (2023.11.17)\n",
      "Requirement already satisfied: click>=7.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from uvicorn>=0.14.0->gradio==3.39) (8.1.7)\n",
      "Collecting h11>=0.8 (from uvicorn>=0.14.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/95/04/ff642e65ad6b90db43e668d70ffb6736436c7ce41fcc549f4e9472234127/h11-0.14.0-py3-none-any.whl (58 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting starlette<0.33.0,>=0.29.0 (from fastapi->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/40/9e/6bfa6be40034fa04cc50e2a81d24a4e5b89279c688b51380d70ac31c0556/starlette-0.32.0.post1-py3-none-any.whl (70 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m70.0/70.0 kB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: anyio in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from httpx->gradio==3.39) (3.7.1)\n",
      "Collecting httpcore==1.* (from httpx->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/56/ba/78b0a99c4da0ff8b0f59defa2f13ca4668189b134bd9840b6202a93d9a0f/httpcore-1.0.2-py3-none-any.whl (76 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: sniffio in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from httpx->gradio==3.39) (1.3.0)\n",
      "Collecting pycparser (from cffi>=1.12->cryptography>=36.0.0->pdfminer.six)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/62/d5/5f610ebe421e85889f2e55e33b7f9a6795bd982198517d912eb1c76e1a53/pycparser-2.21-py2.py3-none-any.whl (118 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m118.7/118.7 kB\u001b[0m \u001b[31m14.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting jsonschema-specifications>=2023.03.6 (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ee/07/44bd408781594c4d0a027666ef27fab1e441b109dc3b76b4f836f8fd04fe/jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB)\n",
      "Collecting referencing>=0.28.4 (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b4/11/d121780c173336c9bc3a5b8240ed31f518957cc22f6311c76259cb0fcf32/referencing-0.32.0-py3-none-any.whl (26 kB)\n",
      "Collecting rpds-py>=0.7.1 (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7d/f9/c6b8d6577c916aa9460dc682b1c7bdffa75d4229904e615e9264f7a4044f/rpds_py-0.16.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m39.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting uc-micro-py (from linkify-it-py<3,>=1->markdown-it-py[linkify]>=2.0.0->gradio==3.39)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d1/1c/5aeb94aa980da111e4fd0c0fbe5ad95ed5bf9bd957f8e2a6178b85ff4da8/uc_micro_py-1.0.2-py3-none-any.whl (6.2 kB)\n",
      "Requirement already satisfied: exceptiongroup in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from anyio->httpx->gradio==3.39) (1.2.0)\n",
      "Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime>=1.7.0->rapidocr_onnxruntime)\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f0/0f/310fb31e39e2d734ccaa2c0fb981ee41f7bd5056ce9bc29b2248bd569169/humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m9.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: mpmath>=0.19 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sympy->onnxruntime>=1.7.0->rapidocr_onnxruntime) (1.3.0)\n",
      "Building wheels for collected packages: ffmpy\n",
      "  Building wheel for ffmpy (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for ffmpy: filename=ffmpy-0.3.1-py3-none-any.whl size=5579 sha256=904c7da850bdddcec43c2866f4ed03c11b1c1baeb8543909e97f4a3788dacd5a\n",
      "  Stored in directory: /home/ma-user/.cache/pip/wheels/fc/1d/38/32e2e8ed0ec76090fa0af8667b849507480cc0c63fa1b69e36\n",
      "Successfully built ffmpy\n",
      "Installing collected packages: pytz, pydub, pyclipper, flatbuffers, ffmpy, websockets, uc-micro-py, tzdata, toolz, Shapely, semantic-version, rpds-py, python-multipart, pyparsing, pycparser, protobuf, pillow, orjson, opencv-python, mdurl, latex2mathml, kiwisolver, humanfriendly, h11, fonttools, cycler, contourpy, aiofiles, uvicorn, starlette, referencing, pdf2image, pandas, mdtex2html, matplotlib, markdown-it-py, linkify-it-py, httpcore, coloredlogs, cffi, onnxruntime, mdit-py-plugins, jsonschema-specifications, httpx, fastapi, cryptography, rapidocr_onnxruntime, pdfminer.six, jsonschema, gradio-client, altair, gradio\n",
      "  Attempting uninstall: pillow\n",
      "    Found existing installation: Pillow 10.1.0\n",
      "    Uninstalling Pillow-10.1.0:\n",
      "      Successfully uninstalled Pillow-10.1.0\n",
      "Successfully installed Shapely-2.0.2 aiofiles-23.2.1 altair-5.2.0 cffi-1.16.0 coloredlogs-15.0.1 contourpy-1.2.0 cryptography-41.0.7 cycler-0.12.1 fastapi-0.108.0 ffmpy-0.3.1 flatbuffers-23.5.26 fonttools-4.47.0 gradio-3.39.0 gradio-client-0.8.0 h11-0.14.0 httpcore-1.0.2 httpx-0.26.0 humanfriendly-10.0 jsonschema-4.20.0 jsonschema-specifications-2023.12.1 kiwisolver-1.4.5 latex2mathml-3.77.0 linkify-it-py-2.0.2 markdown-it-py-2.2.0 matplotlib-3.8.2 mdit-py-plugins-0.3.3 mdtex2html-1.2.0 mdurl-0.1.2 onnxruntime-1.16.3 opencv-python-4.8.1.78 orjson-3.9.10 pandas-2.1.4 pdf2image-1.16.3 pdfminer.six-20231228 pillow-10.0.1 protobuf-4.25.1 pyclipper-1.3.0.post5 pycparser-2.21 pydub-0.25.1 pyparsing-3.1.1 python-multipart-0.0.6 pytz-2023.3.post1 rapidocr_onnxruntime-1.3.9 referencing-0.32.0 rpds-py-0.16.2 semantic-version-2.10.0 starlette-0.32.0.post1 toolz-0.12.0 tzdata-2023.4 uc-micro-py-1.0.2 uvicorn-0.25.0 websockets-11.0.3\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install rapidocr_onnxruntime pdf2image pdfminer.six mdtex2html gradio==3.39 -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "547d64ee-6a7b-4577-a332-23d70e3c0519",
   "metadata": {},
   "source": [
    "## 3. 代码开发"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "06f323cf-b3b6-44d2-be25-cba9a3b3673e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "from langchain.prompts.prompt import PromptTemplate\n",
    "\n",
    "from service.config import LangChainCFG\n",
    "from service.chatglm_service import ChatGLMService\n",
    "# from service.chatglm_service import ChatGLMService,ChatGLMForConditionalGeneration\n",
    "from service.knowledge_service import KnowledgeService\n",
    "\n",
    "\n",
    "class LangChainApplication(object):\n",
    "    #初始化，加载ChatGLM3模型和本地知识库\n",
    "    def __init__(self,config):\n",
    "        self.config = config\n",
    "        self.llm_service = ChatGLMService()\n",
    "        self.llm_service.load_model(model_name_or_path=self.config.llm_model_path)\n",
    "        self.knowledge_service = KnowledgeService(config)\n",
    "    #获取大语言模型返回的答案（基于本地知识库查询）\n",
    "    def get_knowledeg_based_answer(self,query,\n",
    "                                  history_len=5,\n",
    "                                  temperature=0.1,\n",
    "                                  top_p=0.9,\n",
    "                                  top_k=4,\n",
    "                                  chat_history=[]):\n",
    "        #定义查询的提示模板格式：\n",
    "        prompt_template = \"\"\"\n",
    "基于以下已知信息，简洁和专业的来回答用户的问题。\n",
    "如果无法从中得到答案，请说 \"根据已知信息无法回答该问题\" 或 \"没有提供足够的相关信息\"，不允许在答案中添加编造成分，答案请使用中文。\n",
    "已知内容:\n",
    "{context}\n",
    "问题:\n",
    "{question}\n",
    "    \"\"\"\n",
    "        prompt = PromptTemplate(template=prompt_template,\n",
    "                            input_variables=[\"context\", \"question\"])\n",
    "        self.llm_service.history = chat_history[-history_len:] if history_len>0 else []\n",
    "        self.llm_service.temperature = temperature\n",
    "        self.llm_service.top_p = top_p\n",
    "        \n",
    "        #利用预先存在的语言模型、检索器来创建并初始化BaseRetrievalQA类的实例\n",
    "        knowledge_chain = RetrievalQA.from_llm(\n",
    "        llm = self.llm_service,\n",
    "        #基于本地知识库构建一个检索器，并仅返回top_k的结果\n",
    "        retriever = self.knowledge_service.knowledge_base.as_retriever(\n",
    "        search_kwargs={\"k\":top_k}),\n",
    "        prompt = prompt)\n",
    "        #combine_documents_chain的作用是将查询返回的文档内容（page_content）合并到一起作为prompt中context的值\n",
    "        #将combine_documents_chain的合并文档内容改为{page_content}\n",
    "        \n",
    "        knowledge_chain.combine_documents_chain.document_prompt = PromptTemplate(\n",
    "        input_variables=[\"page_content\"],template=\"{page_content}\")\n",
    "        \n",
    "        #返回结果中是否包含源文档\n",
    "        knowledge_chain.return_source_documents = True\n",
    "        \n",
    "        #传入问题内容进行查询\n",
    "        result = knowledge_chain({\"query\":query})\n",
    "        return result\n",
    "    \n",
    "    #获取大语言模型返回的答案（未基于本地知识库查询）\n",
    "    def get_llm_answer(self,query):\n",
    "        result = self.llm_service._call(query)\n",
    "        return result\n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c9e1bf59-5ec2-478f-84d2-91afb2bf3707",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading checkpoint shards: 100%|██████████| 7/7 [00:08<00:00,  1.26s/it]\n",
      "No sentence-transformers model found with name /home/ma-user/work/text2vec-large-chinese. Creating a new one with MEAN pooling.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "result of llm:\n",
      "\n",
      "ModelBox是一个基于Web的建模平台，它允许用户创建、编辑和分享各种类型的模型，包括数据模型、机器学习模型、业务流程模型等。ModelBox旨在为用户提供一个易于使用的界面，以便他们可以快速地创建和管理模型，而无需深入了解代码或复杂的建模技术。通过ModelBox，用户可以轻松地将模型部署到生产环境中，并与其他人共享和协作。ModelBox的主要特点是易于使用、功能丰富、灵活性和可扩展性。\n",
      "\n",
      "#############################################\n",
      "\n",
      "\n",
      "#####init_knowledge_base#####\n",
      "\n",
      "modelbox.jpg\n",
      "modelbox.pdf\n",
      "modelbox.txt\n",
      "第一个应用.md\n",
      "\n",
      "############################################\n",
      "\n",
      "\n",
      "result of knowledge base:\n",
      "\n",
      "{'query': 'ModelBox是什么', 'result': 'ModelBox是一个AI应用开发平台，提供了一种快速、高效、灵活的方式，让开发者能够构建、部署和管理AI应用。ModelBox具有以下特点：\\n\\n1. 基于云的AI平台，可以轻松地进行模型训练和部署。\\n2. 提供了丰富的开发工具和库，包括Python、TensorFlow、PyTorch等热门AI框架。\\n3. 支持多种硬件平台，包括CPU、GPU、TPU等。\\n4. 支持多种操作系统，包括Windows、Linux、macOS等。\\n5. 提供了一套完整的AI应用开发流程，包括数据预处理、模型训练、模型部署、模型优化等。\\n\\nModelBox的应用开发流程如下：\\n\\n1. 确定应用需求，包括输入数据、输出数据、模型类型等。\\n2. 选择合适的开发工具和库，包括Python、TensorFlow、PyTorch等。\\n3. 在ModelBox平台上构建AI应用，包括数据预处理、模型训练、模型部署等。\\n4. 进行模型优化和调试，以提高模型性能和准确度。\\n5. 将模型部署到目标硬件平台上，并进行测试和验证。\\n6. 完成应用开发和部署，并进行维护和更新。\\n\\nModelBox可以帮助开发者快速构建、部署和管理AI应用，提高AI应用的开发效率和性能。', 'source_documents': [Document(page_content='ModelBox特点', metadata={'source': '/home/ma-user/work/docs/modelbox.txt', 'filename': 'modelbox.txt', 'file_directory': '/home/ma-user/work/docs', 'last_modified': '2023-12-29T10:23:20', 'filetype': 'text/plain', 'languages': ['zho', 'kor'], 'category': 'Title'}), Document(page_content='ModelBox解决的问题', metadata={'source': '/home/ma-user/work/docs/modelbox.txt', 'filename': 'modelbox.txt', 'file_directory': '/home/ma-user/work/docs', 'last_modified': '2023-12-29T10:23:20', 'filetype': 'text/plain', 'languages': ['zho', 'kor'], 'category': 'Title'}), Document(page_content='接下来我们给出该应用在ModelBox中的完整开发过程：', metadata={'source': '/home/ma-user/work/docs/第一个应用.md', 'filename': '第一个应用.md', 'file_directory': '/home/ma-user/work/docs', 'last_modified': '2023-12-29T10:23:20', 'filetype': 'text/markdown', 'languages': ['nor', 'eng'], 'page_number': 1, 'category': 'Title'}), Document(page_content='ModelBox AI应用开发——第一个应用', metadata={'source': '/home/ma-user/work/docs/第一个应用.md', 'filename': '第一个应用.md', 'file_directory': '/home/ma-user/work/docs', 'last_modified': '2023-12-29T10:23:20', 'filetype': 'text/markdown', 'languages': ['nor', 'eng'], 'page_number': 1, 'category': 'Title'})]}\n"
     ]
    }
   ],
   "source": [
    "from service.config import LangChainCFG\n",
    "\n",
    "config = LangChainCFG()\n",
    "application = LangChainApplication(config)\n",
    "\n",
    "result = application.get_llm_answer('ModelBox是什么')\n",
    "print('\\nresult of llm:\\n')\n",
    "print(result)\n",
    "print('\\n#############################################\\n')\n",
    "\n",
    "application.knowledge_service.init_knowledge_base()\n",
    "result2 = application.get_knowledeg_based_answer('ModelBox是什么')\n",
    "print('\\n############################################\\n')\n",
    "print('\\nresult of knowledge base:\\n')\n",
    "print(result2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a7937ffc-0494-4714-ad9d-542096a76eec",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading checkpoint shards: 100%|██████████| 7/7 [00:08<00:00,  1.24s/it]\n",
      "No sentence-transformers model found with name /home/ma-user/work/text2vec-large-chinese. Creating a new one with MEAN pooling.\n"
     ]
    }
   ],
   "source": [
    "# 编写Gradio调用函数\n",
    "import mdtex2html\n",
    "from service.config import LangChainCFG\n",
    "from service.configuration_chatglm import ChatGLMConfig\n",
    "\n",
    "\n",
    "#将文本中的字符转为网页上可以支持的字符，避免被误认为是HTML标签\n",
    "def parse_text(text):\n",
    "    lines = text.split(\"\\n\")\n",
    "    lines = [line for line in lines if line != \"\"]\n",
    "    count = 0\n",
    "    for i, line in enumerate(lines):\n",
    "        if \"```\" in line:\n",
    "            count += 1\n",
    "            items = line.split('`')\n",
    "            if count % 2 == 1:\n",
    "                lines[i] = f'<pre><code class=\"language-{items[-1]}\">'\n",
    "            else:\n",
    "                lines[i] = f'<br></code></pre>'\n",
    "        else:\n",
    "            if i > 0:\n",
    "                if count % 2 == 1:\n",
    "                    line = line.replace(\"`\", \"\\`\")\n",
    "                    line = line.replace(\"<\", \"&lt;\")\n",
    "                    line = line.replace(\">\", \"&gt;\")\n",
    "                    line = line.replace(\" \", \"&nbsp;\")\n",
    "                    line = line.replace(\"*\", \"&ast;\")\n",
    "                    line = line.replace(\"_\", \"&lowbar;\")\n",
    "                    line = line.replace(\"-\", \"&#45;\")\n",
    "                    line = line.replace(\".\", \"&#46;\")\n",
    "                    line = line.replace(\"!\", \"&#33;\")\n",
    "                    line = line.replace(\"(\", \"&#40;\")\n",
    "                    line = line.replace(\")\", \"&#41;\")\n",
    "                    line = line.replace(\"$\", \"&#36;\")\n",
    "                lines[i] = \"<br>\"+line\n",
    "    text = \"\".join(lines)\n",
    "    return text\n",
    "\n",
    "#采用流聊天方式（stream_chat）调用模型，使得生成答案有逐字生成的效果\n",
    "# def predict(input, chatbot, max_length, top_p, temperature, history, past_key_values):\n",
    "#     chatbot.append((parse_text(input), parse_text(input)))\n",
    "#     for response, history, past_key_values in model.stream_chat(tokenizer, input, history, past_key_values=past_key_values,\n",
    "#                                                                 return_past_key_values=True,\n",
    "#                                                                 max_length=max_length, top_p=top_p,\n",
    "#                                                                 temperature=temperature):\n",
    "#         chatbot[-1] = (parse_text(input), parse_text(response))\n",
    "\n",
    "#         yield chatbot, history, past_key_values\n",
    "\n",
    "# 在这里定义 application\n",
    "config = LangChainCFG()\n",
    "application = LangChainApplication(config)\n",
    "\n",
    "def predict(input_text, chatbot, max_length, top_p, temperature, history, past_key_values):\n",
    "    application.knowledge_service.init_knowledge_base()\n",
    "    chatbot.append((parse_text(input_text), parse_text(input_text)))\n",
    "    # chat_glm_model = ChatGLMForConditionalGeneration(config=ChatGLMConfig)\n",
    "    \n",
    "    response_dict = application.get_knowledeg_based_answer(parse_text(input_text), history_len=5, temperature=0.1, top_p=0.9, top_k=4, chat_history=history)\n",
    "    if 'result' in response_dict and isinstance(response_dict['result'], str):\n",
    "        result_text = response_dict['result']\n",
    "        response = parse_text(result_text)\n",
    "        chatbot[-1] = (parse_text(input_text), response)\n",
    "        yield chatbot, history, past_key_values\n",
    "    else:\n",
    "        # Handle the case where 'result' key is missing or not a string\n",
    "        # You can modify this part based on your requirements\n",
    "        yield chatbot, history, past_key_values\n",
    "    \n",
    "    # for response, history, past_key_values in chat_glm_model.stream_chat(tokenizer, input, history, past_key_values=past_key_values,\n",
    "    #                                                             return_past_key_values=True,\n",
    "    #                                                             max_length=max_length, top_p=top_p,\n",
    "    #                                                             temperature=temperature):\n",
    "        \n",
    "        \n",
    "\n",
    "#去除输入框的内容\n",
    "def reset_user_input():\n",
    "    return gr.update(value='')\n",
    "\n",
    "#清除状态\n",
    "def reset_state():\n",
    "    return [], [], None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "30c48015-f6de-470f-add7-9ae6c8b40ce0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local URL:  http://127.0.0.1:7860\n",
      "Running on public URL: https://a327a5f4636300ce08.gradio.live\n",
      "\n",
      "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"https://a327a5f4636300ce08.gradio.live\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "#####init_knowledge_base#####\n",
      "\n",
      "modelbox.jpg\n",
      "modelbox.pdf\n",
      "modelbox.txt\n",
      "第一个应用.md\n",
      "\n",
      "#####init_knowledge_base#####\n",
      "\n",
      "modelbox.jpg\n",
      "modelbox.pdf\n",
      "modelbox.txt\n",
      "第一个应用.md\n",
      "\n",
      "#####init_knowledge_base#####\n",
      "\n",
      "modelbox.jpg\n",
      "modelbox.pdf\n",
      "modelbox.txt\n",
      "第一个应用.md\n"
     ]
    }
   ],
   "source": [
    "#运行Gradio界面，运行成功后点击“Running on public URL”后的网页链接即可体验\n",
    "import gradio as gr\n",
    "\n",
    "\n",
    "with gr.Blocks() as demo:\n",
    "    gr.HTML(\"\"\"<h1 align=\"center\">ChatGLM3-6B</h1>\"\"\")\n",
    "\n",
    "    chatbot = gr.Chatbot()\n",
    "    with gr.Row():\n",
    "        with gr.Column(scale=4):\n",
    "            with gr.Column(scale=12):\n",
    "                user_input = gr.Textbox(show_label=False, placeholder=\"Input...\", lines=10,container=False)\n",
    "            with gr.Column(min_width=32, scale=1):\n",
    "                submitBtn = gr.Button(\"Submit\", variant=\"primary\")\n",
    "        with gr.Column(scale=1):\n",
    "            emptyBtn = gr.Button(\"Clear History\")\n",
    "            max_length = gr.Slider(0, 32768, value=8192, step=1.0, label=\"Maximum length\", interactive=True)\n",
    "            top_p = gr.Slider(0, 1, value=0.8, step=0.01, label=\"Top P\", interactive=True)\n",
    "            temperature = gr.Slider(0, 1, value=0.95, step=0.01, label=\"Temperature\", interactive=True)\n",
    "\n",
    "    history = gr.State([])\n",
    "    past_key_values = gr.State(None)\n",
    "\n",
    "    submitBtn.click(predict, [user_input, chatbot, max_length, top_p, temperature, history, past_key_values],\n",
    "                    [chatbot, history, past_key_values], show_progress=True)\n",
    "    submitBtn.click(reset_user_input, [], [user_input])\n",
    "\n",
    "    emptyBtn.click(reset_state, outputs=[chatbot, history, past_key_values], show_progress=True)\n",
    "\n",
    "demo.queue().launch(share=True, inbrowser=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f8f72e0-5534-4b45-8ca0-03de234ab044",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "AIGalleryInfo": {
   "item_id": "0844abfc-df12-4b3c-a02b-08694652b441"
  },
  "flavorInfo": {
   "architecture": "X86_64",
   "category": "GPU"
  },
  "imageInfo": {
   "id": "d996b661-e127-48c4-a90a-fca29535f201",
   "name": "pytorch1.10-cuda10.2-cudnn7-ubuntu18.04"
  },
  "kernelspec": {
   "display_name": "python-3.10.10",
   "language": "python",
   "name": "python-3.10.10"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
